Alber Analafean
121450146
Teknologi Basis Data-RC
Gambar merupakan suatu hal yang sering kita temui diberbagai sisi di dunia ini. Apalagi Gambar merupakan bagian dari multimedia, dalam memperoleh sebuah informasi. Mengikuti perkembangan dunia, ukuran dan jumlah gambar juga ikut bertambah besar juga, hal inilah dibutuhkan teknologi dalam menyimpan gambar tersebut agar bisa disimpan dan diakses secara cepat dan tepat sesuai kebutuhan. Teknolgi Penyimpanan gambar pun, sangat diperlukan dalam berbagai task tertentu, seperti contoh klasifikasi mengunakan CNN serta dalam melakukan analisis sentimen. Sehingga pada laporan ini akan menjelaskann Teknologi database dalam menyimpan gambar pada disk, LMDB dan juga HDFS.
Pada implementasinya, kita mengunakan data CIFAR-10 yang terdiri dari 60.0000 gambar berukuran 32x32 pixel. Walaupun bisa dikataka merupakan data tidak cukup besar. Sehingga perlu dilakukan serialisasi objek untuk menghapus masing-masing dari lima batch.
Environment
Sebelum melakukan, terlebih dahulu install environment:
1.Disk
$ pip install Pillow
2.Lmdb
$ pip install lmdb
3.HDF5
$ pip install h5py
Membuat Path ke direktori
from pathlib import Path
disk_dir = Path("data/disk/")
lmdb_dir = Path("data/lmdb/")
hdf5_dir = Path("data/hdf5/")
disk_dir.mkdir(parents=True, exist_ok=True)
lmdb_dir.mkdir(parents=True, exist_ok=True)
hdf5_dir.mkdir(parents=True, exist_ok=True)
A.Penyimpanan Data
Menyimpan ke disk
from PIL import Image
import csv
def store_single_disk(image, image_id, label):
""" Stores a single image as a .png file on disk.
Parameters:
---------------
image image array, (32, 32, 3) to be stored
image_id integer unique ID for image
label image label
"""
Image.fromarray(image).save(disk_dir / f"{image_id}.png")
with open(disk_dir / f"{image_id}.csv", "wt") as csvfile:
writer = csv.writer(
csvfile, delimiter=" ", quotechar="|", quoting=csv.QUOTE_MINIMAL
)
writer.writerow([label])
Menyimpan ke LMDB
class CIFAR_Image:
def __init__(self, image, label):
# Dimensions of image for reconstruction - not really necessary
# for this dataset, but some datasets may include images of
# varying sizes
self.channels = image.shape[2]
self.size = image.shape[:2]
self.image = image.tobytes()
self.label = label
def get_image(self):
""" Returns the image as a numpy array. """
image = np.frombuffer(self.image, dtype=np.uint8)
return image.reshape(*self.size, self.channels)
Menyimpan satu gambar
import lmdb
import pickle
def store_single_lmdb(image, image_id, label):
""" Stores a single image to a LMDB.
Parameters:
---------------
image image array, (32, 32, 3) to be stored
image_id integer unique ID for image
label image label
"""
map_size = image.nbytes * 10
# Create a new LMDB environment
env = lmdb.open(str(lmdb_dir / f"single_lmdb"), map_size=map_size)
# Start a new write transaction
with env.begin(write=True) as txn:
# All key-value pairs need to be strings
value = CIFAR_Image(image, label)
key = f"{image_id:08}"
txn.put(key.encode("ascii"), pickle.dumps(value))
env.close()
Menyimpan dengan HDF5
import h5py
def store_single_hdf5(image, image_id, label):
""" Stores a single image to an HDF5 file.
Parameters:
---------------
image image array, (32, 32, 3) to be stored
image_id integer unique ID for image
label image label
"""
# Create a new HDF5 file
file = h5py.File(hdf5_dir / f"{image_id}.h5", "w")
# Create a dataset in the file
dataset = file.create_dataset(
"image", np.shape(image), h5py.h5t.STD_U8BE, data=image
)
meta_set = file.create_dataset(
"meta", np.shape(label), h5py.h5t.STD_U8BE, data=label
)
file.close()
Menyimpan Dalam satu gambar
_store_single_funcs = dict(
disk=store_single_disk, lmdb=store_single_lmdb, hdf5=store_single_hdf5
)
Menyimpan Banyak Gambar
Dan sebelumnya kita bisa menlihat implementasi penyimpanan dengan satu gambar, selanjutnya kita menyimpan dengan banyak gambar.
store_many_disk(images, labels):
""" Stores an array of images to disk
Parameters:
---------------
images images array, (N, 32, 32, 3) to be stored
labels labels array, (N, 1) to be stored
"""
num_images = len(images)
# Save all the images one by one
for i, image in enumerate(images):
Image.fromarray(image).save(disk_dir / f"{i}.png")
# Save all the labels to the csv file
with open(disk_dir / f"{num_images}.csv", "w") as csvfile:
writer = csv.writer(
csvfile, delimiter=" ", quotechar="|", quoting=csv.QUOTE_MINIMAL
)
for label in labels:
# This typically would be more than just one value per row
writer.writerow([label])
def store_many_lmdb(images, labels):
""" Stores an array of images to LMDB.
Parameters:
---------------
images images array, (N, 32, 32, 3) to be stored
labels labels array, (N, 1) to be stored
"""
num_images = len(images)
map_size = num_images * images[0].nbytes * 10
# Create a new LMDB DB for all the images
env = lmdb.open(str(lmdb_dir / f"{num_images}_lmdb"), map_size=map_size)
# Same as before — but let's write all the images in a single transaction
with env.begin(write=True) as txn:
for i in range(num_images):
# All key-value pairs need to be Strings
value = CIFAR_Image(images[i], labels[i])
key = f"{i:08}"
txn.put(key.encode("ascii"), pickle.dumps(value))
env.close()
def store_many_hdf5(images, labels):
""" Stores an array of images to HDF5.
Parameters:
---------------
images images array, (N, 32, 32, 3) to be stored
labels labels array, (N, 1) to be stored
"""
num_images = len(images)
# Create a new HDF5 file
file = h5py.File(hdf5_dir / f"{num_images}_many.h5", "w")
# Create a dataset in the file
dataset = file.create_dataset(
"images", np.shape(images), h5py.h5t.STD_U8BE, data=images
)
meta_set = file.create_dataset(
"meta", np.shape(labels), h5py.h5t.STD_U8BE, data=labels
)
file.close()
Mempersiapkan Kumpulan data
Pada proses ini kita mempersiapakan ukuran kumpulan data yang bisa menguji sampai 100.000 gambar.
cutoffs = [10, 100, 1000, 10000, 100000]
# Let's double our images so that we have 100,000
images = np.concatenate((images, images), axis=0)
labels = np.concatenate((labels, labels), axis=0)
# Make sure you actually have 100,000 images and labels
print(np.shape(images))
print(np.shape(labels))
Waktu diperlukan Menyimpan Banyak Gambar
_store_many_funcs = dict(
disk=store_many_disk, lmdb=store_many_lmdb, hdf5=store_many_hdf5
)
from timeit import timeit
store_many_timings = {"disk": [], "lmdb": [], "hdf5": []}
for cutoff in cutoffs:
for method in ("disk", "lmdb", "hdf5"):
t = timeit(
"_store_many_funcs[method](images_, labels_)",
setup="images_=images[:cutoff]; labels_=labels[:cutoff]",
number=1,
globals=globals(),
)
store_many_timings[method].append(t)
# Print out the method, cutoff, and elapsed time
print(f"Method: {method}, Time usage: {t}")
Grafik menunjukkan waktu diperlukan Menyimpan Data
Terlihat setelah dilakukan proses menyimpan banyak gambar, waktu diperlukan HDF5 sangat lambat dibandingkan PNG file dan LMDB. Bisa dikatakan penyimpanan disk & LMDB sangat cocok dalam menyimpan data gambar lebih besar, namun juga bisa dipengaruhi oleh beberapa faktor seperti sistem operasi, perangkat digunakan bahkan ukuran data digunakan.
B.Membaca Data Gambar
Membaca dari disk
def read_single_disk(image_id):
""" Stores a single image to disk.
Parameters:
---------------
image_id integer unique ID for image
Returns:
----------
image image array, (32, 32, 3) to be stored
label associated meta data, int label
"""
image = np.array(Image.open(disk_dir / f"{image_id}.png"))
with open(disk_dir / f"{image_id}.csv", "r") as csvfile:
reader = csv.reader(
csvfile, delimiter=" ", quotechar="|", quoting=csv.QUOTE_MINIMAL
)
label = int(next(reader)[0])
return image, label
Membaca dari LMDB
def read_single_lmdb(image_id):
""" Stores a single image to LMDB.
Parameters:
---------------
image_id integer unique ID for image
Returns:
----------
image image array, (32, 32, 3) to be stored
label associated meta data, int label
"""
# Open the LMDB environment
env = lmdb.open(str(lmdb_dir / f"single_lmdb"), readonly=True)
# Start a new read transaction
with env.begin() as txn:
# Encode the key the same way as we stored it
data = txn.get(f"{image_id:08}".encode("ascii"))
# Remember it's a CIFAR_Image object that is loaded
cifar_image = pickle.loads(data)
# Retrieve the relevant bits
image = cifar_image.get_image()
label = cifar_image.label
env.close()
return image, label
Membaca dari HDF5
def read_single_hdf5(image_id):
""" Stores a single image to HDF5.
Parameters:
---------------
image_id integer unique ID for image
Returns:
----------
image image array, (32, 32, 3) to be stored
label associated meta data, int label
"""
# Open the HDF5 file
file = h5py.File(hdf5_dir / f"{image_id}.h5", "r+")
image = np.array(file["/image"]).astype("uint8")
label = int(np.array(file["/meta"]).astype("uint8"))
return image, label
_read_single_funcs = dict(
disk=read_single_disk, lmdb=read_single_lmdb, hdf5=read_single_hdf5
)
Membaca dari banyak gambar
def read_many_disk(num_images):
""" Reads image from disk.
Parameters:
---------------
num_images number of images to read
Returns:
----------
images images array, (N, 32, 32, 3) to be stored
labels associated meta data, int label (N, 1)
"""
images, labels = [], []
# Loop over all IDs and read each image in one by one
for image_id in range(num_images):
images.append(np.array(Image.open(disk_dir / f"{image_id}.png")))
with open(disk_dir / f"{num_images}.csv", "r") as csvfile:
reader = csv.reader(
csvfile, delimiter=" ", quotechar="|", quoting=csv.QUOTE_MINIMAL
)
for row in reader:
labels.append(int(row[0]))
return images, labels
def read_many_lmdb(num_images):
""" Reads image from LMDB.
Parameters:
---------------
num_images number of images to read
Returns:
----------
images images array, (N, 32, 32, 3) to be stored
labels associated meta data, int label (N, 1)
"""
images, labels = [], []
env = lmdb.open(str(lmdb_dir / f"{num_images}_lmdb"), readonly=True)
# Start a new read transaction
with env.begin() as txn:
# Read all images in one single transaction, with one lock
# We could split this up into multiple transactions if needed
for image_id in range(num_images):
data = txn.get(f"{image_id:08}".encode("ascii"))
# Remember that it's a CIFAR_Image object
# that is stored as the value
cifar_image = pickle.loads(data)
# Retrieve the relevant bits
images.append(cifar_image.get_image())
labels.append(cifar_image.label)
env.close()
return images, labels
def read_many_hdf5(num_images):
""" Reads image from HDF5.
Parameters:
---------------
num_images number of images to read
Returns:
----------
images images array, (N, 32, 32, 3) to be stored
labels associated meta data, int label (N, 1)
"""
images, labels = [], []
# Open the HDF5 file
file = h5py.File(hdf5_dir / f"{num_images}_many.h5", "r+")
images = np.array(file["/images"]).astype("uint8")
labels = np.array(file["/meta"]).astype("uint8")
return images, labels
_read_many_funcs = dict(
disk=read_many_disk, lmdb=read_many_lmdb, hdf5=read_many_hdf5
)
Waktu diperlukan dalam membaca banyak gambar
from timeit import timeit
read_many_timings = {"disk": [], "lmdb": [], "hdf5": []}
for cutoff in cutoffs:
for method in ("disk", "lmdb", "hdf5"):
t = timeit(
"_read_many_funcs[method](num_images)",
setup="num_images=cutoff",
number=1,
globals=globals(),
)
read_many_timings[method].append(t)
# Print out the method, cutoff, and elapsed time
print(f"Method: {method}, No. images: {cutoff}, Time usage: {t}")
Grafik perbedaan waktu dalam membaca penyimpanan data
Dari grafik perbedaan waktu, terlihat bahwa LMDB memiliki waktu lebih cepat daripada HDF5 dan PNG files
Jumlah Memori yang terpakai dala menyimpan data
Dari hasil penyimpanan terlihat bahwa pengunaan memory LMDB lebih besar dibandingkan penyimpanan disk dan HDFS. Pengunaan memori ini juga dipengaruhi oleh beberapa faktor seperti Sistem operasi digunakan, perangat serta ukuran data disimpan sehingga terdapat perbedaan diantara jenis penyimpanan. Walaupun secara waktu LDMB menang namun dalam pengunaan memori HDF5 sangat efisien digunakan
Kesimpulan:
Dari proses menyimpan data gambar serta membaca penyimpanan data gambar, menunjukkan bahwa masing masing
teknologi penyimpanan memiliki kelebihan bahkan kekurangan masing-masing. Dalam segi kecepatan dalam menyimpan LMDB dan PNG file sangat cocok digunakan. Namun dalam sisi pengunaan memori HDF5 dan PNG file sangat cocok dalam mengurangi ruang penyimpanan. Secara skalabilitas penyimpana data cocok mengunakan LMDB dan secara fleksibilitas pengunaan HDF5 sangatlah cocok.
Lampiran:
1.Referensi
https://realpython.com/storing-images-in-python/#storing-to-disk
2.Code:
https://colab.research.google.com/drive/1TbN1rx4OxWqXV4o6_ZSPbxcIQPitfDCr?usp=sharing