4/11/2020 Transparently externalize BLOB to BFILE - Blog dbi services

Infrastructure at your Service


OFFERING  EXPERTISE  TRAINING  JOBS & CAREER  NEWSROOM  ABOUT DBI  CONTACT BLOG 

Transparently externalize BLOB to BFILE     

By Franck Pachot April 12, 2016 Database Administration & Monitoring One Comment

By Franck Pachot
.
Storing documents within the database is the easiest, especially because you get them consistent with the metadata stored in the database. If
you store them externally, then you need to manage their backup, their synchronization to standby site, the consistency in case of flashback or
PITR, etc. However, documents grow (in number and in size thanks to better resolution of scan) and you don’t want a database where half of
the size are documents in read only. If you have no option (partitioning, compression, etc) then you may choose to store the documents
externally. This is usually a complete re-design of the application.
In this blog post, I’ve done a quick test I’ve done to transform some BLOB into External LOB (aka BFILE) and make it transparent to the
application.

It’s just a test of concept. Any comments are welcome if you think something is wrong here.

Some display settings

SQL> set linesize 220 pagesize 1000 echo on

SQL> column filename format a20
SQL> column doc format a80 trunc
SQL> column external_doc format a40 trunc
SQL> whenever sqlerror exit failure;
SQL> connect demo/demo@//localhost/pdb
Connected.

First, I create a table with BLOB

SQL> create table DEMOTAB ( id number, filename varchar2(255),doc blob );

Table created.

And I will fill it with the content of 3 binary files. Let’s take them in $ORACLE_HOME/bin just for the fun of it:

SQL> host ls $ORACLE_HOME/bin | nl | head -3 > /tmp/files.txt

SQL> host cat /tmp/files.txt
1 acfsroot
2 adapters
This website uses cookies to improve your experience. We'll assume
3 adrci you're ok with this, but you can opt-out if you wish. Accept

I’m using SQL*Loader to load them to the BLOB: Reject Read More

https://blog.dbi-services.com/transparently-externalize-blob-to-bfile/ 1/7
4/11/2020 Transparently externalize BLOB to BFILE - Blog dbi services


Infrastructure at your
SQL> host echo "load data Service
infile '/tmp/files.txt' into table DEMOTAB fields terminated by ' ' ( id char(10),filename
SQL>host cat /tmp/sqlldr.ctl
load data infile '/tmp/files.txt' into table DEMOTAB fields terminated by ' ' ( id char(10),filename char(255),doc l
OFFERING  EXPERTISE  TRAINING  JOBS & CAREER  NEWSROOM  ABOUT DBI  CONTACT BLOG 
SQL> host cd $ORACLE_HOME/bin ; sqlldr demo/demo@//localhost/pdb control=/tmp/sqlldr.ctl

SQL*Loader: Release 12.1.0.2.0 on Tue Apr 12 21:03:22 2016

Copyright (c) 1982, 2016, Oracle and/or its affiliates. All rights reserved.

Path used: Conventional

Commit point reached - logical record count 3

Table DEMOTAB:
3 Rows successfully loaded.

Check the log file:

sqlldr.log
for more information about the load.

They are loaded, I can query my table:

SQL> select DEMOTAB.*,dbms_lob.getlength(doc) from DEMOTAB;

ID FILENAME DOC DBMS_LO

---------- -------------------- -------------------------------------------------------------------------------- -------
1 acfsroot 23212F62696E2F7368200A230A230A232061636673726F6F740A23200A2320436F70797269676874
2 adapters 3A0A230A2320244865616465723A206E6574776F726B5F7372632F75746C2F61646170746572732E
3 adrci 7F454C4602010100000000000000000002003E000100000000124000000000004000000000000000

I’m creating a folder to store the files externally, and create a DIRECTORY for it:

SQL> host rm -rf /tmp/files ; mkdir /tmp/files

SQL> create directory DEMODIR as '/tmp/files';
Directory created.

Now I add a BFILE column to my table:

SQL> alter table DEMOTAB add ( external_doc bfile );

Table altered.

My idea is not to move all BLOB to External LOB, but only part of them. For example, old documents can be externalized whereas current ones
stay in the database. That helps to control the database size without taking any risk about consistency in case of PITR.

I’ve there an inline procedure ‘lob_to_file’ that reads a LOB and writes it to a file. In the body of the PL/SQL block I call the procedure for the 2
first rows of my table, and once the files are externalized, I empty the DOC column (the BLOB) and set the EXTERNAL_DOC one (the BFILE):

SQL> set serveroutput on

SQL> declare
tmp_blob blob default empty_blob();
procedure lob_to_file(input_blob in BLOB, file_path in varchar2, file_name in varchar2) as
buffer raw(32767);
buffer_size number:=32767;
amount number;
offset number;
filehandle utl_file.file_type;
blob_size number;
begin
filehandle := utl_file.fopen(file_path, file_name,'wb', 1024);
blob_size:=dbms_lob.getlength(input_blob);
offset:=1;
amount:=32767;
while offset < blob_size loop
This website uses cookies to improveamount,
dbms_lob.read(input_blob, your experience. We'll assume you're ok with this, but you can opt-out
offset, buffer); if you wish. Accept
utl_file.put_raw(filehandle, buffer,true);
offset := offset + buffer_size;
buffer := null;
Reject Read More
end loop;

https://blog.dbi-services.com/transparently-externalize-blob-to-bfile/ 2/7
4/11/2020 Transparently externalize BLOB to BFILE - Blog dbi services
exception when others then

Infrastructure at your Service
utl_file.fclose(filehandle);
raise;

end;
begin
OFFERING
for
 EXPERTISE
c in ( select
 TRAINING
* from DEMOTAB where id <=2JOBS
 & CAREER  NEWSROOM  ABOUT DBI 
) loop CONTACT BLOG 
lob_to_file (c.doc, 'DEMODIR',c.filename);
update DEMOTAB set doc=null,external_doc=bfilename('DEMODIR',c.filename) where id=c.id;
end loop;
end;
/
PL/SQL procedure successfully completed.

Note: don’t take my code as an example. I did it quickly. You should know that best place for code examples is Tim Hall www.oracle-base.com

I can check that I have the two files in my directory

SQL> host ls -l /tmp/files

total 128
-rw-r--r--. 1 oracle oinstall 945 Apr 12 21:03 acfsroot
-rw-r--r--. 1 oracle oinstall 13360 Apr 12 21:03 adapters

and compare it to the size of original file:

SQL> host ls -l $ORACLE_HOME/bin | head -4

total 644308
-rwxr-xr-x. 1 oracle oinstall 945 May 24 2014 acfsroot
-rwxr-xr-x. 1 oracle oinstall 13360 Mar 23 2015 adapters
-rwxr-x--x. 1 oracle oinstall 46156 Mar 25 17:20 adrci

And here is my table:

SQL> select id,filename,dbms_lob.getlength(doc),external_doc from DEMOTAB;

ID FILENAME DBMS_LOB.GETLENGTH(DOC) EXTERNAL_DOC

---------- -------------------- ----------------------- ----------------------------------------
1 acfsroot bfilename('DEMODIR', 'acfsroot')
2 adapters bfilename('DEMODIR', 'adapters')
3 adrci 46156 bfilename(NULL)

You see that first two rows have empty BLOB but a BFILE addressing the files in DEMODIR
The third row is untouched.

Now, my idea is to make it transparent for the application, so I create a view on it which transparently retrieves the External LOB when LOB is
null:

SQL> create view DEMOVIEW as select id,filename,nvl(doc,external_doc) doc from DEMOTAB;

View created.

And now time to query. The application does a select into a BLOB so let’s do the same:

SQL> variable doc blob;

SQL> exec select doc into :doc from DEMOVIEW where id=1;
PL/SQL procedure successfully completed.
SQL> print doc

DOC
--------------------------------------------------------------------------------
23212F62696E2F7368200A230A230A232061636673726F6F740A23200A2320436F70797269676874

This is the LOB coming from the external file. I get it as a BLOB when I query the view.
This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept

And now querying the one that is still stored in the database: Reject Read More

https://blog.dbi-services.com/transparently-externalize-blob-to-bfile/ 3/7
4/11/2020 Transparently externalize BLOB to BFILE - Blog dbi services


Infrastructure at your Service
SQL> exec select doc into :doc from DEMOVIEW where id=3;
 procedure successfully completed.
PL/SQL
SQL> print doc 
OFFERING EXPERTISE
  TRAINING JOBS & CAREER  NEWSROOM  ABOUT DBI  CONTACT BLOG 
DOC
--------------------------------------------------------------------------------
7F454C4602010100000000000000000002003E000100000000124000000000004000000000000000

Querying the view instead of the table (and you can play with synonyms for that) the application get the document without knowing wheter it
comes from the database or the external directory. It seems that externalizing binary documents do not require a re-design of the application.

One Comment

Rainer Stenzel says: Reply to Rainer

July 24, 2016 at 9 h 29 min

Hello Franck,
do you know whether there are already BFILE cloud adapters available or ahead to take this approach one step further?
Best regards,
Rainer Stenzel

Leave a Reply
My comment is..

Name *

Email *

Website

Save myuses
This website name, email,to
cookies and websiteyour
improve in this browser forWe'll
experience. the next time you're
assume I comment.
ok with this, but you can opt-out if you wish. Accept

Reject Read More

SUBMIT COMMENT

https://blog.dbi-services.com/transparently-externalize-blob-to-bfile/ 4/7
4/11/2020 Transparently externalize BLOB to BFILE - Blog dbi services


Infrastructure at your Service

This site uses Akismet to reduce
OFFERING spam.Learn
EXPERTISE how yourcomment
TRAINING JOBS data is processed.
& CAREER  NEWSROOM  ABOUT DBI  CONTACT BLOG 

Post view(s) : 7,584

Search... 

CATEGORIES

Application integration & Middleware

AWS

Big Data

Business Intelligence

Cloud

Control-M

Database Administration & Monitoring

Database management

Development & Performance

DevOps

Docker

Entreprise content management

Hardware & Storage

Kubernetes

MariaDB

MySQL

NoSQL

Operation systems

Oracle

Postgres

Security

SQL Server

Technology Survey

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish.
Unclassified Accept

Reject Read More

https://blog.dbi-services.com/transparently-externalize-blob-to-bfile/ 5/7
4/11/2020 Transparently externalize BLOB to BFILE - Blog dbi services

RECENT ARTICLES
Infrastructure at your Service 


Getting started with Exasol – Distribution keys
OFFERING  EXPERTISE  TRAINING  JOBS & CAREER  NEWSROOM  ABOUT DBI  CONTACT BLOG 

Jenkins how to create a cyclic job

Control M/ EM Send Mail and Alert Window notification with HHMMSS format with a job

Control-M/EM Logon issue when connecting to the CCM Cannot get UTF8 locale

Control-M/EM CTM5323 security protection violation , delete folder XXXXXX

TAG CLOUD

12.2 Alfresco AlwaysOn Availability groups AWS CDB Cloud Cluster database Data Guard DBA Docker Documentum enterprisedb
High availability Installation Linux Linux/UNIX Microsoft Migration Monitoring multitenant ODA Oracle
Oracle 11g
Oracle 12c Oracle 18c Oracle Enterprise Manager Oracle Enterprise Manager Cloud 12c Oracle OpenWorld PDB Performance Pluggable
Databases PostgreSQL PowerShell RMAN Security SQL SQL Server SQL Server 2008 SQL Server 2012 SQL Server 2014
SQL Server 2016 Troubleshooting Upgrade

BLOG ROLL

Florian Haas' blog

Oracle Scratchpad - Jonathan Lewis' blog

The Tom Kyte Blog (Ask Tom)

Blog of Adar-Consult

Alex Gorbachev's blog

Marcus Mönnig's Oracle & Mumbai Blog

Franck Pachot
Principal Consultant / Database Evangelist
Oracle ACE Director
Oracle Database OCM 12c certified
AWS Database Specialty certified
Oak Table member

RSS for this blog: feed

Twitter: @FranckPachot
LinkedIn : www.linkedin.com/in/franckpachot
Podcast en français: DBPod
This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept

Reject Read More

https://blog.dbi-services.com/transparently-externalize-blob-to-bfile/ 6/7
4/11/2020 Transparently externalize BLOB to BFILE - Blog dbi services


Infrastructure at your Service

OFFERING  EXPERTISE  TRAINING  JOBS & CAREER  NEWSROOM  ABOUT DBI  CONTACT BLOG 
EXPERTISE IN DATABASE & MIDDLEWARE

Oracle database expertise

SQL Server expertise
MySQL/MariaDB expertise
PostgreSQL expertise
NoSQL expertise
SharePoint expertise
OpenText Documentum expertise
Linux expertise (Oracle Linux, Red Hat)

TRAININGS IN DATABASE & MIDDLEWARE

Microsoft
Oracle
Open Source DB
Operating system

USEFUL INFORMATION

News & Events

Jobs openings
Offices
Blog of dbi services
Imprint

© 2015 dbi services sa

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept

Reject Read More

https://blog.dbi-services.com/transparently-externalize-blob-to-bfile/ 7/7