LeoFS - A Storage System for a Data Lake and the Web
LeoFS: a highly available, distributed, eventually consistent object/blob store
Overview

LeoFS is a highly available, distributed, eventually consistent object/blob store. If you are searching a storage system that is able to store huge amount and various kind of files such as photo, movie, log data and so on, LeoFS is suitable for that.

LeoFS is supporting the following features:

Multi Protocol S3-API Support LeoFS is an Amazon S3 compatible storage system. Switch to LeoFS to decrease your cost from more expensive public-cloud solutions. REST-API Support To easily access LeoFS with REST-API NFS Support NFS support was provided from LeoFS v1.1, the current status of which is beta. Large Object Support LeoFS can handle files with more than GB Multi Data Center Replication LeoFS is a highly scalable, fault-tolerant distributed file system without SPOF. LeoFS's cluster can be viewed as ONE-HUGE storage. It consists of a set of loosely connected nodes. We can build a global scale storage system with easy operations Architecture
LeoFS: a highly available, distributed, eventually consistent object/blob store

LeoFS consists of 3 applications -LeoFS Storage,LeoFS Gateway andLeoFS Manager which depend on Erlang.

LeoFS Gateway handles http-request and http-response from any clients when using REST-API OR S3-API. Also, it is already built in the object-cache mechanism (memory and disk cache).

LeoFS Storage handles GET, PUT and DELETE objects as well as metadata. Also, it has replicator, recoverer and queueing mechanism in order to keep running a storage node and realise eventual consistency.

LeoFS Manager always monitors LeoFS Gateway and LeoFS Storage nodes. The main monitoring status are Node status and RING’s checksum in order to realise to keep high availability and keep data consistency.

We can access LeoFS server using S3 clients and S3 client libries of each programming language .

Slide

The presentation - Scaling and High Performance Storage System: LeoFS was given at Erlang User Conference 2014 in Stockholm on June 2014

GOALs LeoFS aims to provide all of 3-HIGHs as follow: HIGH Reliability Nine nines - Operating ratios is 99.9999999% High Scalability Build huge-cluster at low cost HIGH Cost Performance Fast - Over 10Gbps A lower cost than other storage Provide easy management and easy operation Further Reference LeoFS Documentation . Build LeoFS with LeoFS Packages

LeoFS packages have been already provided on the Web. You're able to easily install LeoFS on your environments.

LeoProject CentOS-7.0 CentOS-6.x 14.04 and later Community FreeBSD

Here is the installation manual.

Build LeoFS From Source (For Developers)

Here, we explain how to build LeoFS from source code.

First, you have to install the following packages to build Erlang and LeoFS.

## [CentOS] $ sudo yum install libuuid-devel cmake check check-devel ## [Ubuntu] $ sudo apt-get install build-essential libtool libncurses5-dev libssl-dev cmake check

Then, install Erlang.

## ## 1. Install libatomic ## $ wget http://www.hpl.hp.com/research/linux/atomic_ops/download/libatomic_ops-7.2d.tar.gz $ tar xzvf libatomic_ops-7.2d.tar.gz $ cd libatomic_ops-7.2 $ ./configure --prefix=/usr/local $ make $ sudo make install ## ## 2. Install Erlang (17.5) ## $ wget http://www.erlang.org/download/otp_src_17.5.tar.gz $ tar xzf otp_src_17.5.tar.gz $ cd otp_src_17.5 $ ./configure --prefix=/usr/local/erlang/17.5 \ --enable-smp-support \ --enable-m64-build \ --enable-halfword-emulator \ --enable-kernel-poll \ --without-javac \ --disable-native-libs \ --disable-hipe \ --disable-sctp \ --enable-threads \ --with-libatomic_ops=/usr/local $ make $ sudo make install ## ## 3. Set PATH ## $ vi ~/.profile ## append the follows: export ERL_HOME=/usr/local/erlang/17.5 export PATH=$PATH:$ERL_HOME/bin $ source ~/.profile

Then, clone source of LeoFS and libraries from GitHub.

$ git clone https://github.com/leo-project/leofs.git $ cd leofs $ git checkout -b develop remotes/origin/develop $ ./rebar get-deps $ ./git_checkout.sh develop

Then, build LeoFS with the following commands.

$ make clean $ make $ make release

Now, you can find the LeoFS package as follow.

$ ls package/ leo_gateway/ leo_manager_0/ leo_manager_1/ leo_storage/ README.md

Then, we can start and access LeoFS with the following commands. Also, you're able to easily operate LeoFS withleofs-adm script.

$ package/leo_manager_0/bin/leo_manager start $ package/leo_manager_1/bin/leo_manager start $ package/leo_storage/bin/leo_storage start $ package/leo_gateway/bin/leo_gateway start $ ./leofs-adm status [System Confiuration] -----------------------------------+---------- Item | Value -----------------------------------+---------- Basic/Consistency level -----------------------------------+---------- system version | 1.2.16 cluster Id | leofs_1 DC Id | dc_1 Total replicas | 2 number of successes of R | 1 number of successes of W | 1 number of successes of D | 1 number of rack-awareness replicas | 0 ring size | 2^128 -----------------------------------+---------- Multi DC replication settings -----------------------------------+---------- max number of joinable DCs | 2 number of replicas a DC | 1 -----------------------------------+---------- Manager RING hash -----------------------------------+---------- current ring-hash | 3923d007 previous ring-hash | 3923d007 -----------------------------------+---------- [State of Node(s)] -------+--------------------------+--------------+----------------+----------------+---------------------------- type | node | state | current ring | prev ring | updated at -------+--------------------------+--------------+----------------+----------------+---------------------------- S | [email protected] | running | 3923d007 | 3923d007 | 2015-10-30 09:44:11 +0900 -------+--------------------------+--------------+----------------+----------------+---------------------------- $ ./leofs-adm start OK $ ./leofs-adm status -----------------------------------+---------- Item | Value -----------------------------------+---------- Basic/Consistency level -----------------------------------+---------- system version | 1.2.16 cluster Id | leofs_1 DC Id | dc_1 Total replicas | 2 number of successes of R | 1 number of successes of W | 1 number of successes of D | 1 number of rack-awareness replicas | 0 ring size | 2^128 -----------------------------------+---------- Multi DC replication settings -----------------------------------+---------- max number of joinable DCs | 2 number of replicas a DC | 1 -----------------------------------+---------- Manager RING hash -----------------------------------+---------- current ring-hash | 3923d007 previous ring-hash | 3923d007 -----------------------------------+---------- [State of Node(s)] -------+--------------------------+--------------+----------------+----------------+---------------------------- type | node | state | current ring | prev ring | updated at -------+--------------------------+--------------+----------------+----------------+---------------------------- S | [email protected] | running | 3923d007 | 3923d007 | 2015-10-30 09:44:11 +0900 G | [email protected] | running | 3923d007 | 3923d007 | 2015-10-30 09:45:27 +0900 -------+--------------------------+--------------+----------------+----------------+---------------------------- Build a LeoFS Cluster

You can easily build a LeoFS cluster.

Please refer here .

Configure LeoFS

About the configuration of LeoFS, please refer here .

Benchmarking

You can benchmark LeoFS withBasho Bench.

Here is a documentation to benchmark LeoFS.

Integration Test

We're able to easily check LeoFS withleofs_test whether LeoFS has issues or not before getting installed LeoFS in your dev/staging/production environment(s).

Milestones Version 1 DONE -v1.0 Multi Data Center Replication Increase compatibility S3-APIs#5 Other bucket operations DONE -v1.1 NFS v3 Support (alpha) Improve Web GUI Console (Option) DONE -v1.2 NFS v3 Support (beta) Watchdog Auto-compaction WIP - v1.3 NFS v3 Support (stable) Hinted Hand-off (v1.3.2) Improve compatibility S3-APIs#6 AWS Signature v4 support (v1.3.0) Custom Metadata Support (v1.3.1) v1.4 Erasure Code LeoManager's cluster replaces from mnesia dependensy to another implementation Improve compatibility S3-APIs#7 Object Encryption Version 2 v2.0 NFS v3 Support (stable) Improve performance of the list objects, the ls command Improvement of the Multi Data Center Replication Searching objects by a custom-metadata Improve compatibility S3-APIs#8 Objects Expiration into a Bucket Object Versioning Improve Web GUI console,LeoFS Center (option) v2.2 NFS v4 Support Data Deduplication Improve compatibility S3-APIs#8 Sponsors

LeoProject/LeoFS is sponsored by Rakuten, Inc. and supported by Rakuten Institute of Technology .

本文系统(linux)相关术语:linux系统 鸟哥的linux私房菜 linux命令大全 linux操作系统

主题: ErlangRESTScalaGitGitHubUbuntuHIGUTADMInc.
分页:12
转载请注明
本文标题:LeoFS: a highly available, distributed, eventually consistent object/blob store
本站链接:http://www.codesec.net/view/520921.html
分享请点击:


1.凡CodeSecTeam转载的文章,均出自其它媒体或其他官网介绍,目的在于传递更多的信息,并不代表本站赞同其观点和其真实性负责;
2.转载的文章仅代表原创作者观点,与本站无关。其原创性以及文中陈述文字和内容未经本站证实,本站对该文以及其中全部或者部分内容、文字的真实性、完整性、及时性,不作出任何保证或承若;
3.如本站转载稿涉及版权等问题,请作者及时联系本站,我们会及时处理。
登录后可拥有收藏文章、关注作者等权限...
技术大类 技术大类 | 系统(linux) | 评论(0) | 阅读(117)