{"id":17590,"date":"2019-12-04T05:39:34","date_gmt":"2019-12-03T20:39:34","guid":{"rendered":"http:\/\/www.skyarch.net\/blog\/?p=17590"},"modified":"2019-12-04T06:02:31","modified_gmt":"2019-12-03T21:02:31","slug":"cmy305-iot-big-data-pipeline-for-serverless-analytics-and-anomaly-detection%e3%82%bb%e3%83%83%e3%82%b7%e3%83%a7%e3%83%b3%e3%82%92%e5%8f%97%e8%ac%9b%e3%81%97%e3%81%be%e3%81%97%e3%81%9f","status":"publish","type":"post","link":"https:\/\/www.skyarch.net\/blog\/cmy305-iot-big-data-pipeline-for-serverless-analytics-and-anomaly-detection%e3%82%bb%e3%83%83%e3%82%b7%e3%83%a7%e3%83%b3%e3%82%92%e5%8f%97%e8%ac%9b%e3%81%97%e3%81%be%e3%81%97%e3%81%9f\/","title":{"rendered":"\u3010AWS re:Invent2019\u3011CMY305 &#8211; IoT big data pipeline for serverless analytics and anomaly detection\u30bb\u30c3\u30b7\u30e7\u30f3\u3092\u53d7\u8b1b\u3057\u307e\u3057\u305f"},"content":{"rendered":"<ul>\n<li><a href=\"#overview\">\u6982\u8981<\/a><\/li>\n<li><a href=\"#reflink\">\u53c2\u8003\u30ea\u30f3\u30af<\/a><\/li>\n<li><a href=\"#session\">\u30bb\u30c3\u30b7\u30e7\u30f3\u306b\u3064\u3044\u3066<\/a><\/li>\n<\/ul>\n<h2 id=\"overview\">\u6982\u8981<\/h2>\n<p>IoT big data pipeline for serverless analytics and anomaly detection\u30bb\u30c3\u30b7\u30e7\u30f3\u3092\u53d7\u8b1b\u3057\u307e\u3057\u305f<\/p>\n<p>\u8b1b\u6f14\u8005: Gabriel Ram\u00edrez<br \/>\n<a href=\"https:\/\/aws.amazon.com\/jp\/developer\/community\/heroes\/gabriel-ramirez\/\">https:\/\/aws.amazon.com\/jp\/developer\/community\/heroes\/gabriel-ramirez\/<\/a><\/p>\n<p>\u30bb\u30c3\u30b7\u30e7\u30f3\u306f\u5927\u304d\u304f\u4e0b\u8a182\u3064\u306e\u30d1\u30fc\u30c8\u306b\u5206\u304b\u308c\u8a71\u3055\u308c\u3066\u3044\u307e\u3057\u305f\u3002<\/p>\n<ul>\n<li>StreamingDataProcessiong<\/li>\n<li>\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3<\/li>\n<li>\u30c7\u30e21\n<ul>\n<li>KinesisDataPipeline\u3092\u4f5c\u6210<\/li>\n<li>KinesisDataGenerator\u3092\u5229\u7528\u3057\u3066\u30c7\u30fc\u30bf\u3092\u6d41\u3057\u8fbc\u307fS3\u3078\u4fdd\u5b58\u51fa\u6765\u3066\u3044\u308b\u304b\u78ba\u8a8d    <\/li>\n<\/ul>\n<\/li>\n<li>\u30c7\u30e22\n<ul>\n<li>AWS Connected Vehicle Solution\u306b\u3064\u3044\u3066\u5b9f\u969b\u306b\u52d5\u3044\u3066\u3044\u308b\u52d5\u753b\u306e\u7d39\u4ecb<br \/>\n\u8eca\u306e\u30a8\u30f3\u30b8\u30f3\u3092On\u306b\u3057\u3066...\u5834\u6240\u3084\u901f\u5ea6\u306e\u30c7\u30fc\u30bf\u304c\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u306b\u8fd1\u3044\u304b\u305f\u3061\u3067\u98db\u3093\u3067\u304f\u308b\u69d8\u5b50<br \/>\n\u307e\u305f\u3001GeoFencing\u3092\u8a2d\u5b9a\u3057\u3066\u3001\u7279\u5b9a\u533a\u57df\u304b\u3089\u51fa\u305f\u3089\u30a2\u30e9\u30fc\u30c8\u3092\u9cf4\u3089\u3059\u7b49\u3092\u898b\u308b\u4e8b\u304c\u3067\u304d\u307e\u3057\u305f\u3002<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h3 id=\"reflink\">\u53c2\u8003\u30ea\u30f3\u30af<\/h3>\n<ul>\n<li>KinesisDataGenerator\u306e\u5229\u7528\u65b9\u6cd5<br \/>\n<a href=\"https:\/\/aws.amazon.com\/jp\/blogs\/news\/test-your-streaming-data-solution-with-the-new-amazon-kinesis-data-generator\/\">https:\/\/aws.amazon.com\/jp\/blogs\/news\/test-your-streaming-data-solution-with-the-new-amazon-kinesis-data-generator\/<\/a><\/li>\n<li>AWS Connected Vehicle Solution<br \/>\n<a href=\"https:\/\/docs.aws.amazon.com\/ja_jp\/solutions\/latest\/connected-vehicle-solution\/overview.html\">https:\/\/docs.aws.amazon.com\/ja_jp\/solutions\/latest\/connected-vehicle-solution\/overview.html<\/a><\/li>\n<\/ul>\n<h2 id=\"session\">\u30bb\u30c3\u30b7\u30e7\u30f3\u306b\u3064\u3044\u3066<\/h2>\n<h3>StreamingDataProcessing\u3067\u554f\u984c\u306b\u306a\u308b\u4e8b<\/h3>\n<ul>\n<li>Retry<\/li>\n<li>Too many requests<\/li>\n<li>Throttling<\/li>\n<li>ServerOutOfMemory<\/li>\n<\/ul>\n<h3>\u3069\u306e\u3088\u3046\u306a\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u3067\u30af\u30ea\u30a2\u3059\u308b\u304b<\/h3>\n<h4>EC2 Pattern<\/h4>\n<img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173453-1024x768.jpg\" alt=\"\" width=\"728\" height=\"546\" class=\"aligncenter size-large wp-image-17593\" srcset=\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173453.jpg 1024w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173453-300x225.jpg 300w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173453-768x576.jpg 768w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173453-728x546.jpg 728w\" sizes=\"auto, (max-width: 728px) 100vw, 728px\" \/>\n<h4>Serverless Pattern<\/h4>\n<img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173526-1024x768.jpg\" alt=\"\" width=\"728\" height=\"546\" class=\"aligncenter size-large wp-image-17592\" srcset=\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173526.jpg 1024w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173526-300x225.jpg 300w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173526-768x576.jpg 768w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173526-728x546.jpg 728w\" sizes=\"auto, (max-width: 728px) 100vw, 728px\" \/>\n<h4>ReliableServerless Pattern<\/h4>\n<p>\u9593\u306b\u30ad\u30e5\u30fc\u3092\u631f\u3080\u4e8b\u306b\u3088\u308a...<br \/>\n<img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173600-1024x768.jpg\" alt=\"\" width=\"728\" height=\"546\" class=\"aligncenter size-large wp-image-17591\" srcset=\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173600.jpg 1024w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173600-300x225.jpg 300w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173600-768x576.jpg 768w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_173600-728x546.jpg 728w\" sizes=\"auto, (max-width: 728px) 100vw, 728px\" \/><\/p>\n<h4>Streaming and Real-time data systems<\/h4>\n<p>\u30b9\u30c8\u30ea\u30fc\u30df\u30f3\u30b0\u30c7\u30fc\u30bf\u3092\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u306b\u51e6\u7406\u3059\u308b\u305f\u3081\u306b\u306f Amazon Kinesis \u3092\u5229\u7528\u3059\u308b\u4e8b\u304c\u30d9\u30b9\u30c8<br \/>\nKinesis\u306e\u57fa\u672c\u7684\u306a\u6982\u5ff5\u3067\u3042\u308bThroughput\u3084Shard\u8a2d\u5b9a\u7b49\u306b\u3064\u3044\u3066\u89e6\u308c\u305f\u4e0a\u3067Kinesis\u306eAnormalyDetect (RANDOM_CUT_FOREST) \u306b\u95a2\u3057\u3066\u89e6\u308c\u3089\u308c\u3066\u3044\u307e\u3057\u305f\u3002<\/p>\n<img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_175019-1024x768.jpg\" alt=\"\" width=\"728\" height=\"546\" class=\"aligncenter size-large wp-image-17596\" srcset=\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_175019.jpg 1024w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_175019-300x225.jpg 300w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_175019-768x576.jpg 768w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_175019-728x546.jpg 728w\" sizes=\"auto, (max-width: 728px) 100vw, 728px\" \/>\n<p>\u4f8b: \u30b9\u30c8\u30ea\u30fc\u30e0\u3067\u30c7\u30fc\u30bf\u306e\u7570\u5e38\u3092\u691c\u51fa\u3059\u308b (RANDOM_CUT_FOREST \u95a2\u6570)<br \/>\n<a href=\"https:\/\/docs.aws.amazon.com\/ja_jp\/kinesisanalytics\/latest\/dev\/app-anomaly-detection.html\">https:\/\/docs.aws.amazon.com\/ja_jp\/kinesisanalytics\/latest\/dev\/app-anomaly-detection.html<\/a><\/p>\n<h2>AWS Connected Vehicle Solution\u3067\u306e\u5229\u7528\u4f8b<\/h2>\n<p>\u8eca\u304b\u3089\u767a\u4fe1\u3055\u308c\u305f\u60c5\u5831\u304c\u3001AWS IoT\u7d4c\u7531\u3067\u5165\u3063\u3066\u304d\u3066\u305d\u306e\u5f8c\u3001\u3069\u306e\u3088\u3046\u306b\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u306b\u4fdd\u7ba1\u3055\u308c\u308b\u304b\uff08\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u3067\u7570\u5e38\u691c\u77e5\u3092\u5b9f\u65bd\u3057\u305f\u4e0a\u3067\uff09<br \/>\n\u3068\u3044\u3046\u4e8b\u304c\u8a18\u8f09\u3055\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u30c7\u30e2\u52d5\u753b\u304c\u9762\u767d\u304b\u3063\u305f\u306e\u3067\u3059\u304c\u3001\u898b\u3064\u3051\u3089\u308c\u305a...<\/p>\n<img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_180330-1024x768.jpg\" alt=\"\" width=\"728\" height=\"546\" class=\"aligncenter size-large wp-image-17594\" srcset=\"https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_180330.jpg 1024w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_180330-300x225.jpg 300w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_180330-768x576.jpg 768w, https:\/\/www.skyarch.net\/blog\/wp-content\/uploads\/2019\/12\/IMG_20191202_180330-728x546.jpg 728w\" sizes=\"auto, (max-width: 728px) 100vw, 728px\" \/>\n<p>\u8a73\u7d30\u306f\u4e0b\u8a18\u30ea\u30f3\u30af\u5148\u306b\u8a73\u3057\u304f\u8a18\u8f09\u3055\u308c\u3066\u3044\u307e\u3057\u305f<br \/>\n<a href=\"https:\/\/docs.aws.amazon.com\/solutions\/latest\/connected-vehicle-solution\/architecture.html\">https:\/\/docs.aws.amazon.com\/solutions\/latest\/connected-vehicle-solution\/architecture.html<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u6982\u8981 \u53c2\u8003\u30ea\u30f3\u30af \u30bb\u30c3\u30b7\u30e7\u30f3\u306b\u3064\u3044\u3066 \u6982\u8981 IoT big data pipeline for serverless analytics and anomaly detection\u30bb\u30c3\u30b7\u30e7\u30f3\u3092\u53d7\u8b1b\u3057\u307e\u3057\u305f \u8b1b\u6f14\u8005: Ga&#8230;<\/p>\n","protected":false},"author":2,"featured_media":17587,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_locale":"ja","_original_post":"17590","footnotes":""},"categories":[923],"tags":[],"class_list":{"0":"post-17590","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-aws-reinvent-2019","8":"ja"},"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.skyarch.net\/blog\/wp-json\/wp\/v2\/posts\/17590","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.skyarch.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.skyarch.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.skyarch.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.skyarch.net\/blog\/wp-json\/wp\/v2\/comments?post=17590"}],"version-history":[{"count":10,"href":"https:\/\/www.skyarch.net\/blog\/wp-json\/wp\/v2\/posts\/17590\/revisions"}],"predecessor-version":[{"id":17613,"href":"https:\/\/www.skyarch.net\/blog\/wp-json\/wp\/v2\/posts\/17590\/revisions\/17613"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.skyarch.net\/blog\/wp-json\/wp\/v2\/media\/17587"}],"wp:attachment":[{"href":"https:\/\/www.skyarch.net\/blog\/wp-json\/wp\/v2\/media?parent=17590"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.skyarch.net\/blog\/wp-json\/wp\/v2\/categories?post=17590"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.skyarch.net\/blog\/wp-json\/wp\/v2\/tags?post=17590"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}