Learning Journal

 https://docs.google.com/document/d/11DCwE8qZ6wI9fruTsrTjsAxQWBqivJPGwgNvQ1wmjek/edit


Comments

Post a Comment

Popular posts from this blog

Designing Data Intensive Applications - Ch 1 - Reliability, Scalability, Maintainability

Reliability Scalability  Maintainability Data Engines as developers see it; database, ccaches, search index, stream processing, batch processing. How the data is distributed in disks encoding data Reliability - anticipate faults and can tolerate them. Faults - component failure. failure - system failure. hardware faults. software faults. human errors.  telemetry using interfaces, decoupling, testing Scalability -  Load parameters - request to web server - read or write to database - cache hit rate - active users could be average case or small number of extreme cases Twitter example : 4.6K to max 12k writes per user, but 300K reads per user. So work is done pushing writes to individual users caches at write time so read time can be faster. write times become a challenge when they involve so much leg work. still done within 5 seconds. Now twitter does a hybrid model where most tweets follow above approacch, but celebrity tweets are sent at read time. Performance : throughpu...
Read more

Udacity - Data Engineering - Cloud Data Warehousing

 So I've roughly navigated this course and it was quite a challenge going through the theory because once the architectures were discussed it was just a whole bunch of clicking through in AWS and that can get boring. Architecture : For the architecture itself, the idea is a back office and a front office. The back office is all the sources and individual processes that bring in data. The Data warehouse simplifies these schemas into (possibly) a star model and then makes it easier and faster to use for the analytics / BI division (front office).  There are a few variants, where BI can directly access the main source, where DW can be department specific, or unique for each department still maintaining integrity among common columns. Cloud / AWS : Doing this in the cloud provides quicker start time, elasticity, scalability. The general idea is to read from sources and move to a staging S3 bucket and then push to a DW. For smaller tables it might be possible to directly use an EC2...
Read more

Udacity - Data Engineering - Intro

 This is a challenging course I embarked on two years back and could not complete at that time. I'm back here in an attempt to improve my scope of skills, to improve my mental stamina and give myself something that can make me feel accomplished. This has 4 sections - Data Modeling - Cloud Data Warehousing - Spark and Data Lakes - Automating Data Pipelines There are 4 challenging projects to each of these sections and a final capstone project combining them. The technologies that span these courses are SQL, NOSQL, basic python, AWS, Spark, Airflow.
Read more