{"id":2643,"date":"2018-01-30T16:29:44","date_gmt":"2018-01-30T16:29:44","guid":{"rendered":"https:\/\/devbloglavaprotocols.nityo.in\/learning-about-data-processing-on-google-cloud\/"},"modified":"2018-01-30T16:29:44","modified_gmt":"2018-01-30T16:29:44","slug":"learning-about-data-processing-on-google-cloud","status":"publish","type":"post","link":"https:\/\/www.lavaprotocols.com\/the-cloud-blog\/learning-about-data-processing-on-google-cloud\/","title":{"rendered":"Learning about Data Processing on Google Cloud"},"content":{"rendered":"<div style=\"margin-top: 0px; margin-bottom: 0px;\" class=\"sharethis-inline-share-buttons\" ><\/div><p><span style=\"font-weight: 400;\">Everyone\u2019s talking about big data. But do companies actually know how to extract relevant and clean data to derive meaningful insights? <\/span><\/p>\n<p><!--more--><\/p>\n<p><span style=\"font-weight: 400;\">We wanted to equip our developers with the right knowledge about data processing on Google Cloud. So, we sent them for a four-day workshop, \u2018Data Engineering on Google Cloud Platform\u2019.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Held from Jan 30 to Feb 2, the workshop provided participants a hands-on experience on how to:<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">design and build data processing systems;<\/span><\/li>\n<li><span style=\"font-weight: 400;\">build end-to-end data pipelines;<\/span><\/li>\n<li><span style=\"font-weight: 400;\">analyse data;<\/span><\/li>\n<li><span style=\"font-weight: 400;\">and carry out machine learning;<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">on the Google Cloud Platform. <\/span><\/p>\n<p><span style=\"font-weight: 400;\">The workshop also featured presentations and demos and covered structured, unstructured, and streaming data.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The modules covered were as follows:<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Module 1: Google Cloud Dataproc Overview<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Module 2: Running Dataproc Jobs<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Module 3: Integrating Dataproc with Google Cloud Platform<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Module 4: Making Sense of Unstructured Data with Google\u2019s Machine Learning APIs<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Module 5: Serverless data analysis with BigQuery<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Module 6: Serverless, autoscaling data pipelines with Dataflow<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Module 7: Getting started with Machine Learning<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Module 8: Building ML models with Tensorflow<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Module 9: Scaling ML models with CloudML<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Module 10: Feature Engineering<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Module 11: Architecture of streaming analytics pipelines<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Module 12: Ingesting Variable Volumes<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Module 13: Implementing streaming pipelines<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Module 14: Streaming analytics and dashboards<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Module 15: High throughput and low-latency with Bigtable<\/span><\/li>\n<\/ul>\n<div id=\"attachment_10053\" style=\"width: 827px\" class=\"wp-caption aligncenter\">\n <img decoding=\"async\" aria-describedby=\"caption-attachment-10053\" data-attachment-id=\"10053\" data-permalink=\"https:\/\/www.lavaprotocols.com\/2018\/01\/30\/data-processing-on-google-cloud\/img_2009\/\" data-orig-file=\"https:\/\/i0.wp.com\/www.lavaprotocols.com\/wp-content\/uploads\/2018\/08\/IMG_2009-e1533201008854.jpg?fit=1008%2C953&#038;ssl=1\" data-orig-size=\"1008,953\" data-comments-opened=\"0\" data-image-meta=\"{\"aperture\":\"2.2\",\"credit\":\"\",\"camera\":\"iPhone 6\",\"caption\":\"\",\"created_timestamp\":\"1517477314\",\"copyright\":\"\",\"focal_length\":\"2.65\",\"iso\":\"250\",\"shutter_speed\":\"0.033333333333333\",\"title\":\"\",\"orientation\":\"3\"}\" data-image-title=\"IMG_2009\" data-image-description=\"\" data-image-caption=\"\" data-medium-file=\"https:\/\/i0.wp.com\/www.lavaprotocols.com\/wp-content\/uploads\/2018\/08\/IMG_2009-e1533201008854.jpg?fit=300%2C284&#038;ssl=1\" data-large-file=\"https:\/\/i0.wp.com\/www.lavaprotocols.com\/wp-content\/uploads\/2018\/08\/IMG_2009-e1533201008854.jpg?fit=1024%2C768&#038;ssl=1\" loading=\"lazy\" class=\"wp-image-10053\" src=\"https:\/\/4843500.fs1.hubspotusercontent-na1.net\/hubfs\/4843500\/Imported_Blog_Media\/IMG_2009-e1533196772804-1024x768.jpg\" alt=\"\" width=\"817\" height=\"610\" data-recalc-dims=\"1\"><\/p>\n<p id=\"caption-attachment-10053\" class=\"wp-caption-text\"><em>Our developers Amiruddin (left) and Dzarief (right)<\/em><\/p>\n<\/div>\n<p>\u00a0<\/p>\n<div id=\"attachment_10054\" style=\"width: 832px\" class=\"wp-caption aligncenter\">\n <img decoding=\"async\" aria-describedby=\"caption-attachment-10054\" data-attachment-id=\"10054\" data-permalink=\"https:\/\/www.lavaprotocols.com\/2018\/01\/30\/data-processing-on-google-cloud\/img_2013\/\" data-orig-file=\"https:\/\/i0.wp.com\/www.lavaprotocols.com\/wp-content\/uploads\/2018\/08\/IMG_2013.jpg?fit=3264%2C2448&#038;ssl=1\" data-orig-size=\"3264,2448\" data-comments-opened=\"0\" data-image-meta=\"{\"aperture\":\"2.2\",\"credit\":\"\",\"camera\":\"iPhone 6\",\"caption\":\"\",\"created_timestamp\":\"1517478609\",\"copyright\":\"\",\"focal_length\":\"4.15\",\"iso\":\"32\",\"shutter_speed\":\"0.033333333333333\",\"title\":\"\",\"orientation\":\"1\"}\" data-image-title=\"IMG_2013\" data-image-description=\"\" data-image-caption=\"\" data-medium-file=\"https:\/\/i0.wp.com\/www.lavaprotocols.com\/wp-content\/uploads\/2018\/08\/IMG_2013.jpg?fit=300%2C225&#038;ssl=1\" data-large-file=\"https:\/\/i0.wp.com\/www.lavaprotocols.com\/wp-content\/uploads\/2018\/08\/IMG_2013.jpg?fit=1024%2C768&#038;ssl=1\" loading=\"lazy\" class=\"wp-image-10054\" src=\"https:\/\/4843500.fs1.hubspotusercontent-na1.net\/hubfs\/4843500\/Imported_Blog_Media\/IMG_2013-1024x768.jpg\" alt=\"\" width=\"822\" height=\"616\" srcset=\"https:\/\/4843500.fs1.hubspotusercontent-na1.net\/hubfs\/4843500\/Imported_Blog_Media\/IMG_2013-Aug-04-2022-07-44-40-49-AM.jpg 1024w, https:\/\/4843500.fs1.hubspotusercontent-na1.net\/hubfs\/4843500\/Imported_Blog_Media\/IMG_2013-1.jpg 300w, https:\/\/4843500.fs1.hubspotusercontent-na1.net\/hubfs\/4843500\/Imported_Blog_Media\/IMG_2013-3.jpg 768w, https:\/\/4843500.fs1.hubspotusercontent-na1.net\/hubfs\/4843500\/Imported_Blog_Media\/IMG_2013-Aug-04-2022-07-44-41-53-AM.jpg 610w, https:\/\/4843500.fs1.hubspotusercontent-na1.net\/hubfs\/4843500\/Imported_Blog_Media\/IMG_2013-2.jpg 510w, https:\/\/4843500.fs1.hubspotusercontent-na1.net\/hubfs\/4843500\/Imported_Blog_Media\/IMG_2013-Aug-04-2022-07-44-41-07-AM.jpg 1080w, https:\/\/4843500.fs1.hubspotusercontent-na1.net\/hubfs\/4843500\/Imported_Blog_Media\/IMG_2013-4.jpg 2160w, https:\/\/4843500.fs1.hubspotusercontent-na1.net\/hubfs\/4843500\/Imported_Blog_Media\/IMG_2013.jpg 3240w\" sizes=\"auto, (max-width: 822px) 100vw, 822px\" data-recalc-dims=\"1\"><\/p>\n<p id=\"caption-attachment-10054\" class=\"wp-caption-text\"><em>Workshop in session.<\/em><\/p>\n<\/div>\n<p>\u00a0<\/p>\n<p><i><span style=\"font-weight: 400;\">Lava is an authorised Cloud Partner of Google and is a <\/span><\/i><a href=\"https:\/\/lavaprotocols.com\/google-apps-for-work\/\"><i><span style=\"font-weight: 400;\">reseller of G Suite<\/span><\/i><\/a><i><span style=\"font-weight: 400;\"> (previously known as Google Apps, Google Maps for Work, and Google Cloud Platform) in Malaysia. With more than a decade of experience in the industry, we\u2019re proud to say we\u2019re one of the leading cloud consultants and service providers in the Asia Pacific region.<\/span><\/i><\/p>\n<p><script type=\"text\/javascript\">\npiAId = '418952';\npiCId = '56152';\npiHostname = 'pi.pardot.com';<\/p>\n<p>(function() {\n\tfunction async_load(){\n\t\tvar s = document.createElement('script'); s.type = 'text\/javascript';\n\t\ts.src = ('https:' == document.location.protocol ? 'https:\/\/pi' : 'http:\/\/cdn') + '.pardot.com\/pd.js';\n\t\tvar c = document.getElementsByTagName('script')[0]; c.parentNode.insertBefore(s, c);\n\t}\n\tif(window.attachEvent) { window.attachEvent('onload', async_load); }\n\telse { window.addEventListener('load', async_load, false); }\n})();\n<\/script><\/p>\n<p><span class=\"et_bloom_bottom_trigger\"><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>We wanted to equip our developers with the right knowledge about data processing on Google Cloud. So, we sent them for a four-day workshop.<\/p>\n","protected":false},"author":1,"featured_media":2644,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[53],"class_list":["post-2643","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","tag-events"],"jetpack_featured_media_url":"https:\/\/www.lavaprotocols.com\/the-cloud-blog\/wp-content\/uploads\/2024\/10\/IMG_1998-scaled.jpg","_links":{"self":[{"href":"https:\/\/www.lavaprotocols.com\/the-cloud-blog\/wp-json\/wp\/v2\/posts\/2643","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lavaprotocols.com\/the-cloud-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lavaprotocols.com\/the-cloud-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lavaprotocols.com\/the-cloud-blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lavaprotocols.com\/the-cloud-blog\/wp-json\/wp\/v2\/comments?post=2643"}],"version-history":[{"count":0,"href":"https:\/\/www.lavaprotocols.com\/the-cloud-blog\/wp-json\/wp\/v2\/posts\/2643\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.lavaprotocols.com\/the-cloud-blog\/wp-json\/wp\/v2\/media\/2644"}],"wp:attachment":[{"href":"https:\/\/www.lavaprotocols.com\/the-cloud-blog\/wp-json\/wp\/v2\/media?parent=2643"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lavaprotocols.com\/the-cloud-blog\/wp-json\/wp\/v2\/categories?post=2643"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lavaprotocols.com\/the-cloud-blog\/wp-json\/wp\/v2\/tags?post=2643"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}