Managing Storage:: Trends, Challenges, and Options (2007-2008)
Managing Storage:: Trends, Challenges, and Options (2007-2008)
Managing Storage:: Trends, Challenges, and Options (2007-2008)
Managing Storage:
Trends, Challenges, and Options (2007–2008)
Poorly designed or managed storage infrastructures put the entire business at risk in the
event of a catastrophic failure. A robust storage infrastructure requires highly reliable equip-
ment as well as a strong team of experts to manage it efficiently.
This paper focuses on storage infrastructure and presents findings from a global survey
of more than 1,200 IT professionals. This study will help IT managers benchmark their
plans to the overall trends in the industry. The first such study was published in 2006 and
a large number of IT and storage managers used the information to refine their planning
and decision-making.
Key Challenges
IT and storage managers and storage professionals across companies of all sizes face the
following mission-critical challenges:
Explosive growth in storage requirements and a widening storage technology knowledge gap
across the industry are making all of the above mission-critical tasks even more challenging.
Nearly all critical data is now stored on external disk storage subsystems. The average
usable capacity is approximately 222 TB which is typically spread across multiple sites.
Growth in storage requirements, larger capacity disks and subsystems, and affordable
pricing have all led to large storage configurations. Over 45 percent of responding compa-
nies now have more than 100 TB of usable storage to manage.
Storage subsystems, SANs, and backup/recovery technologies are most commonly imple-
mented, followed by replication, NAS, and DAS technologies. IP SAN and CAS technologies
have started to emerge in these companies.
Each of these storage technology segments is unique, offering its own specific business and
operational value. Each requires a different set of skills for effective design and manage-
ment. Lack of knowledge and expertise in a specific segment can lead to under-deployment
of one or more of these technologies.
Storage groups are responsible for overall planning, design, implementation, monitoring,
administering, managing, and operations. While the structure of the group, titles, and roles
may not be standardized, responsibilities and tasks are common across companies.
A strong correlation was found between the installed storage capacity and the size of the
storage group. The ratio is high at lower capacities, and it reduces as the installed capacity
grows. One storage professional is deployed to manage every 20 TB (1:20) of usable storage
for installations having up to 100 TB of usable storage. At 500 TB, the ratio reduces to 1:40.
The shortage of experienced storage professionals and the lack of storage technology edu-
cation in the marketplace and in academics have restricted the growth of information storage
and management functions. EMC has taken the lead and has initiated storage technology
education by collaborating with several leading universities and IT training companies.
The following key initiatives offer options for storage managers and professionals to acquire
or improve their skills to benefit their organizations:
Education Solutions
• ‘Open’ Storage Technology Curriculum
– Unique offering in the industry; leads with concepts and principles
– Covers all segments of information storage and management technology
All of the above education solutions are available globally via EMC® Education Services to
EMC customers, partners, and employees.
In an attempt to help address the widening knowledge gap in the industry, the following
exclusive programs were introduced to enable non-EMC users, as well as university stu-
dents, to take advantage of the ‘open’ storage technology curriculum to build a successful
career in this high-growth industry:
This unprecedented explosion of data, its increasing criticality, and business’ dependency
on digitized information are leading to larger and more complex storage environments that
are increasingly challenging to manage. From the perspective of data availability and pro-
tection, information storage infrastructure is the most critical component of an overall
IT infrastructure. It plays a critical role in making applications work efficiently, both locally
and across multiple sites. With the increasing complexity and criticality of storage, highly
skilled and focused storage groups are as mission-critical as the technology being deployed.
This paper summarizes a global research study that was conducted to learn how companies
are meeting these challenging requirements. These findings will assist IT and storage man-
agers to compare and correlate their plans with the overall trends in the industry. Even
though each company has unique requirements, this information will be helpful in building
stronger and more-efficient storage management teams. Stronger storage management
teams will, in turn, lead to more-robust storage infrastructures.
The first such study was carried out by EMC in 2005–2006 and was found to be very useful
by IT and storage managers globally. The updates and revisions include:
• Most current information via global survey of over 1,200 managers (15 percent)
and storage professionals (85 percent)
• Further details related to storage groups such as job titles, tasks and responsibilities,
and hiring plans
• Technical environments
• Management challenges
The study was carried out between December 2006 and February 2007. We used compre-
hensive surveys and reached out to thousands of storage professionals to assemble and
compile this information.
• EMC users as well as those using storage solutions from other vendors
Over $1B
31%
Asia-Pacific
Japan
14%
Each of these activities is ongoing at various levels in each of the companies. Activities
such as backup/recovery have been in practice for decades; still the professionals believe
that they are not doing enough or not performing them well. The following table summa-
rizes the input from managers and professionals. There is a strong synergy between the
managers and individual contributors as they have identified exactly the same challenges
in similar priority order.
Explosive growth in storage requirements and the storage professionals’ knowledge and
skill gaps are the primary reasons for not executing many of these activities to the desired
levels. These gaps are not necessarily due to lack of competence, but to the fact that a com-
prehensive storage technology education has not been available. For example, colleges
and universities have not, until now, included storage technology in their courses. On the
other hand, vendor training typically focuses on their products covering deployment and
usage rather than building skills and competency to architect, design, integrate, and man-
age entire infrastructure and end-to-end information lifecycle.
We will learn the methodology by which most storage professionals acquire knowledge and
build skills to carry out their assignments in section 4 of this paper.
• Nearly 45 percent of the companies have 100 TB or more usable storage to manage.
Up to 100TB
55%
100–500 TB
Two 20%
Three 26%
11%
• Storage subsystems, by default, are the most important segment as they provide the
backbone infrastructure, storage capacity, reliability, availability, performance, and
connectivity.
• Two segments, storage areas networks (SANs) and backup/recovery (BR), were rated
important by more than 75 percent of the storage professionals.
• NAS and local replication are considered important by 25–50 percent of the
professionals.
• CAS, IP-SAN, and emerging technologies are considered important by 10–25 percent of
the professionals.
Each of the technology segments is unique, bringing its own specific business or operational
values. For example, SAN and NAS provide connectivity options with unique functionality,
while BR and replication technologies provide options for information protection against
planned and unplanned outages.
100%
80%
% Respondents
60%
40%
20%
0%
SAN Backup/ Remote NAS Local CAS IP-SAN
Recovery Repl. Repl.
80%
% Respondents
60%
40%
20%
0%
SAN Backup/ NAS Repli- DAS IP-SAN CAS
Recovery cation
These technology segments (Figures 5 and 6) are at various stages of evolution and maturity.
Each is considered sophisticated and complex, and each requires unique skills to assess,
plan, design, deploy, and manage them effectively. Deploying specialized experts within
the teams, dedicated to their specialty segments, is the most effective way to manage such
diverse technology.
Job titles and descriptions of dedicated storage professionals are evolving. The following
are the most common job functions being deployed by the studied organizations:
Storage Administrators
BR Administrators
Storage Architects
BC Adminstrators
Others
4.1 Responsibilities
Storage groups are responsible for the overall planning, design, implementation, monitoring,
managing, testing, and operating all components of the infrastructure. Interactions with
IT and storage managers and professionals resulted in the list of activities/tasks for which
they are responsible (Table 3).
The list includes various job functions, including storage administration, architects, DR
admin., BR admin., etc. Percent time captured for each of the activities highlights the effort
involved and possible importance of the tasks. This list could be used as a tool to define
responsibilities of the storage group and individuals.
40%
20%
0%
SAN Backup/ Storage NAS Remote Local CAS IP-SAN
Recovery Sub- Repli. Repli.
systems
The ratio of installed capacity to the number of professionals in the storage team is high at
lower capacities and it reduces as the capacity increases. If we count 100 percent dedicated
storage professionals (if two individuals are spending 50 percent of their time on storage-
related activities, they are counted as one professional in this exercise), the current ratio at
100 TB is 1:20 (one professional for every 20 TB installed). In other words, five fulltime, ded-
icated professionals are managing a storage pool of 100 TB and its associated applications.
The ratio reached 1:40 at 500 TB, where approx. 12–13 professionals are managing 500 TB.
In larger storage infrastructures, the ratio continues to reduce. Typically, 17 professionals
are managing 1 PB of storage (ratio nearly 1:60).
1600
Installed Usable Storage Capacity (TB)
1400
1200
1000
800
600
400
200
0
2 3 4 5 12 17 25
# Professionals dedicated to managing storage infrastructure
This is a key challenge for storage managers because it underscores the very real skills gap
in their teams. Sub-optimal skills yield sub-optimal storage deployment. On the other hand,
a well-skilled team will lead to higher productivity, better deployment and management of
technology, and optimization of the number of professionals required.
Figure 10 below illustrates the overall rating of the storage teams against the identified
tasks and responsibilities (Table 3). Strong, moderate, and weak bars indicate the level of
expertise within the overall storage team in the represented companies.
SAN
B/R
Subsystems
NAS
Remote Repl.
Local Repl.
CAS
IP SAN
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Figure 11 maps skill sets of the storage teams against the important technology segments.
Skill levels of the teams in SAN, backup/recovery, and storage subsystems are rated high,
while skills in NAS, remote replication, local replication, etc. are rated low. A correlation
between ability to execute tasks (as shown in Figure 10) and competence in relevant tech-
nology segment (from Figure 11) will paint a clear picture of the effective competence of the
storage group. Detailed assessment of each individual within the group is required to
ascertain strengths and weaknesses for each task and related technology segment.
Considering the aggressive hiring requirements and plans, the lack of skilled resources
becomes a serious bottleneck. Figure 12 highlights the level of hiring requirements across
the industry. The hiring plans indicate a growth of two to three times the number of exist-
ing staff are required to be hired, trained, and deployed.
BR Administrators 126%
Figure 13 shows the preferred hiring options. The majority of managers prefer to hire expe-
rienced professionals to reduce the learning period and reduce risks associated with hiring
new employees. The next best alternative is to hire well-trained and certified individuals.
Hire Experienced
Hire Certified
Internal Appointment
Engage Consultants
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
% of Managers
With the shortage of skilled manpower in the industry—capable, experienced, skilled indi-
viduals are usually not available for hiring. Major factors for this skills shortage include lack
of storage technology education in the marketplace and in academia. EMC has taken the lead
and successfully introduced storage technology curriculum (and certification) in several
universities as well as in the open market via public classes. A lot more needs to be done
to convert this industry-wide bottleneck into lucrative employment opportunities for aspir-
ing professionals.
On-the-job Training
Vendor Training
Self Development
Certification Training
IT Training Companies
In-house Training
At the very least, three distinct requirements can be identified (represented as Skills Pyramid
in Figure 15).
Foundations: Required for all members of the team. Strong understanding of underlying
technical concepts across all storage technology segments is a must to be able to:
- Align to assignments
Specialization based on deployed and responsibilities
technology and assignments - In-depth skills in selected
technology segment(s)
- Technology concepts
Build strong foundation & principles
across all storage technology segments - All storage technology
segments
Formal specialization assignments also will help address the development needs for nearly
two-thirds of the existing workforce.
This study underscores a widening knowledge and skill gap in this mission-critical industry.
A very aggressive hiring requirement is possibly the most important challenge faced by the
managers today, which becomes even more challenging considering the shortage of expe-
rienced or certified professionals.
Nearly two-thirds of the storage professionals employed today require additional knowl-
edge and skills to perform their responsibilities efficiently. This is an important revelation.
• Companies without formal and focused storage management groups must evaluate this
mission-critical requirement.
• Companies with dedicated storage teams must carefully analyze skills requirements
and current skill levels in their teams.
• Storage technology vendors should develop knowledge and skills in the industry when
they introduce new technologies.
• Leading universities, colleges, and training providers must include storage technology
courses in their curricula to offer their graduates career opportunities in this high
growth industry. The next generation of IT professionals, or anyone looking for a
different career path, has a great opportunity to learn the skills and meet the demands
in this high-growth, dynamic environment.
The following key initiatives offer options for storage managers and professionals to
acquire or improve their skills to benefit their organizations:
Education Solutions
• ‘Open’ Storage Technology Curriculum
– Unique offering in the industry; leads with concepts and principles
– Covers all segments of information storage and management technology
– Adapted by several universities
– Public classes by independent training providers for non-EMC users
All of the above education solutions are available globally via EMC Education Services to
EMC customers, partners, and employees.
In an attempt to help address the widening knowledge gap in the industry, the following
exclusive programs have been introduced which enable non-EMC users as well as univer-
sity students to take advantage of ‘open’ storage technology curriculum and to build a suc-
cessful career in this high-growth industry.
• Introduced in mid-2006, the program has helped establish alliances with a large
number of universities in several countries.
For more information on the programs, offerings, alliances, and partnerships, visit
www.EMC.com/training.
Alok Shrivastava
Senior Director, Education Services
EMC Global Services
EMC Corporation
[email protected]
Online:
www.EMC.com/training
http://education.EMC.com
E-mail:
[email protected]
Phone:
888-EMC-TRNG (888-362-8764)
International:
[email protected]
+44 208 758 6080 (UK)
+49 6196 4728 666 (Germany)
[email protected]
+61 2 9922 7888 (ANZ)
+65 6333 6200 (South Asia)
[email protected]
+81 3 3345 5900 (Japan)
[email protected]
+82 22125 7503 (Korea)
EMC Corporation
Hopkinton
Massachusetts
01748-9103
1-508-435-1000
In North America 1-866-464-7381
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS
PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
EMC2, EMC, and where information lives are registered trademarks and EMC Proven is a trademark of EMC Corporation.
All other trademarks used herein are the property of their respective owners.
EMC Perspective
H2159.2
www.EMC.com