Any plans on roadmap to add support for DPDK blobfs/blobstore

[aley1]

Hi PhotonLib Authors,

Just want to commend you guys for adding DPDK support via F-Stack. This does make life much easier for us who want high-performance networking but may not have the know how / time to integrate these libraries.

Will like to know if there are plans to add support for DPDK nvme driver via blobfs/blobstore to read/write append only files?

[lihuiba]

What a coincidence! We began studying this specific issue just an hour ago. And we'll come back when we have an answer.

[lihuiba]

@aley1
BTW, what kind of applications are you developing with photon? Just for curiosity.

[aley1]

@lihuiba
I am working on a log/event processing and storage application. Ingesting high volume of data, transforming and storing outputs from these data. Part of the stack includes rocksdb so Photonlib + DPDK network/nvme is an ideal fit. Also need to read large files from disk so I presume blobfs can improve performance

[beef9999]

@lihuiba I am working on a log/event processing and storage application. Ingesting high volume of data, transforming and storing outputs from these data. Part of the stack includes rocksdb so Photonlib + DPDK network/nvme is an ideal fit. Also need to read large files from disk so I presume blobfs can improve performance

Is it an open-source project?

[aley1]

No, it's not.

[lihuiba]

includes rocksdb

I assume that you know we have already ported RocksDB to photon, do you?

[lihuiba]

read large files from disk

And we'll also support user-space ext4, both read and write.

[aley1]

includes rocksdb

I assume that you know we have already ported RocksDB to photon, do you?

Not yet but will definitely do it later. Currently development is still early stage and done on standard rocksdb

read large files from disk

And we'll also support user-space ext4, both read and write.

Ok great, hopefully it has same or close performance to bypassing filesystem but less complexity

[lihuiba]

Not yet but will definitely do it later.

You can checkout our work and see whether it works.

hopefully it has same or close performance

I'm not clear about its performance. We are doing it primarily for compatibility with kernel storage.

[aley1]

Hope you guys will give SPDK blobfs serious consideration as well.

[lihuiba]

Hope you guys will give SPDK blobfs serious consideration as well.

Yes, we'll integrate them as long as it's not difficult.

[lihuiba]

Hope you guys will give SPDK blobfs serious consideration as well.

@aley1
We have released our integration of user-space ext4 in https://github.com/alibaba/PhotonLibOS/tree/main/fs/extfs.

And we are unfortunately running short of hands these days, so we can not work on SPDK and blobfs at present.
As we have already integrated DPDK, I suppose SPDK and blobfs are not too difficult. Can you make a pull request(s) for them?

[aley1]

I may not be able to attempt to do it this year, as there are other priorities, but may do so in early 2024.

[lihuiba]

That's cool!. All kinds of contributions are welcome!