Scribd
Upload
90 views18 pages

Google File System (GFS)

The Google File System (GFS) was designed to address the needs of Google's massive data storage and high query load. GFS uses a master-chunk server architecture with replication to provide scalability, fault tolerance, and high performance. It operates on a shared storage cluster and allows applications to read, write, and append files in parallel via simple operations. GFS's use of chunking, replication and load balancing enables it to scale efficiently across thousands of commodity servers.

Uploaded by

Mohit Gautam
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
Download as pdf or txt
90 views18 pages

Google File System (GFS)

The Google File System (GFS) was designed to address the needs of Google's massive data storage and high query load. GFS uses a master-chunk server architecture with replication to provide scalability, fault tolerance, and high performance. It operates on a shared storage cluster and allows applications to read, write, and append files in parallel via simple operations. GFS's use of chunking, replication and load balancing enables it to scale efficiently across thousands of commodity servers.

Uploaded by

Mohit Gautam
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
Download as pdf or txt
Download as pdf or txt
You are on page 1/ 18

GOOGLE FILE SYSTEM (GFS)

Hans Vatne Hansen


Distributed File Systems

● Andrew File System (AFS)


● Network File System (NFS)
● Coda
● Microsoft Distributed File System (DFS)
● Apple Filing Protocol (AFP)
Distributed File Systems

Andrew File System Coda


● Performance CODA = AFS +
● Scalability ● Disconnected operation
● Availability ● For mobile computing
● Security
● For partial network failures
● Network bandwidth
adaptation
● Client side caching
● Server replication
Motivation for GFS
● Nothing is small in Google land ● Clusters all over the world
● Peta-bytes of data
● Thousands of queries
● Millions of users
served per second
● Lots of services and servers
→ Scalability ● One query reads
● Failures are normal hundreds of MB of data
● Network connections ● One query consumes
● Hard disks billions of CPU cycles
● Power supplies
→ Fault tolerance
● Monitoring and maintenance is hard
● A distributed, fault-tolerant
file system is needed!
→ Autonomic computing
Google Data Centers

● Scaling out on commodity hardware is cheaper than


scaling up on high-end servers
● Google servers:
● > 15 000 servers (2003)
● ~ 200 000 (2005)
● ~ 1 M servers (2010)

● Data centers are composed of standard shipping


containers with 1160 servers in each
Google Data Centers
Google Data Centers
Chunks and Chunk Servers

Chunk
● Similar to block in file systems
● Size is always 64 MB
● Less fragmentation
● Eases management
● Sent directly to clients
Master Servers

Master Server
● Coordinates cluster
● Updates operation log
● Stores meta-data
Master Server – Chunk Server Communication

State updates
● Is a chunk server down?
● Are there disk failures on a chunk server?
● Are any replicas corrupted?
● Which chunk replicas does a chunk server store?

Instructions
● Create new chunk
● Delete existing chunks
GFS Architecture
Read operation
Write operation (1/2)
Write operation (2/2)
Record Append
● Record append allows multiple clients to append data to the same file concurrently while guaranteeing atomicity

Algorithm
● Application originates record append request.
● GFS client translates request and sends it to master.
● Master responds with chunk handle and (primary + secondary) replica locations.
● Client pushes write data to all locations.
● Primary check if record fits in specified chunk.
● If record does not fit, then the primary:
● Pads the chunk.
● Tells secondaries to do the same.
● Informs the client.
● Client retries append with the next chunk.
● If records fits, then the primary:
● Appends the record.
● Tells secondaries to do the same.
● Receives responses from secondaries.
● Send final response to the client.
Fault Tolerance

● Master and chunk server recovers extremely fast


● Chunks, operation log and master state is replicated
● Replication is done across multiple machines and data
centers in case of severe failures
Conclusions

● GFS has
● Performance
● Scalability
● Fault-tolerance

● GFS is
● Easy to maintain
● Cheapest solution for Google

● Clients and applications can


● Read in parallel
● Write in parallel
● Append in parallel
References

● Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung,


The Google File System, ACM Symposium on Operating
Systems Principles, 2003
● Naushad UzZaman, Survey on Google File System,
CSC 456 (Operating Systems), 2007
● Jonathan Strickland, How the Google File System
Works, HowStuffWorks.com, 2010
● Wikipedia Contributors, Google File System, Wikipedia -
The Free Encyclopedia, 2010

You might also like