meerkat package

Subpackages

Submodules

meerkat.config module

class DatasetsConfig(_root_dir: 'str' = '/home/docs/.meerkat/datasets')[source]

Bases: object

property root_dir
class DisplayConfig(max_rows: 'int' = 10, show_images: 'bool' = True, max_image_height: 'int' = 128, max_image_width: 'int' = 128, show_audio: 'bool' = True)[source]

Bases: object

max_image_height: int = 128
max_image_width: int = 128
max_rows: int = 10
show_audio: bool = True
show_images: bool = True
class MeerkatConfig(display: 'DisplayConfig', datasets: 'DatasetsConfig')[source]

Bases: object

classmethod from_yaml(path: Optional[str] = None)[source]
datasets: DatasetsConfig
display: DisplayConfig

meerkat.datapanel module

DataPanel class.

class DataPanel(data: Optional[Union[dict, list]] = None, *args, **kwargs)[source]

Bases: CloneableMixin, FunctionInspectorMixin, LambdaMixin, MappableMixin, MaterializationMixin, ProvenanceMixin

Meerkat DataPanel class.

add_column(name: str, data: Union[Sequence, ndarray, Series, Tensor], overwrite=False) → None[source]

Add a column to the DataPanel.

append(dp: DataPanel, axis: Union[str, int] = 'rows', suffixes: Tuple[str] = None, overwrite: bool = False) → DataPanel[source]

Append a batch of data to the dataset.

example_or_batch must have the same columns as the dataset (regardless of what columns are visible).

batch(batch_size: int = 1, drop_last_batch: bool = False, num_workers: int = 0, materialize: bool = True, shuffle: bool = False, *args, **kwargs)[source]

Batch the dataset. TODO:

Parameters

  • batch_size – integer batch size

  • drop_last_batch – drop the last batch if its smaller than batch_size

Returns

batches of data

consolidate()[source]
filter(function: Optional[Callable] = None, with_indices=False, input_columns: Optional[Union[str, List[str]]] = None, is_batched_fn: bool = False, batch_size: Optional[int] = 1, drop_last_batch: bool = False, num_workers: int = 0, materialize: bool = True, pbar: bool = False, **kwargs) → Optional[DataPanel][source]

Filter operation on the DataPanel.

classmethod from_arrow(table: Table)[source]

Create a Dataset from a pandas DataFrame.

classmethod from_batch(batch: Dict[str, Union[List, AbstractColumn]]) → DataPanel[source]

Convert a batch to a Dataset.

classmethod from_batches(batches: Sequence[Dict[str, Union[List, AbstractColumn]]]) → DataPanel[source]

Convert a list of batches to a dataset.

classmethod from_csv(filepath: str, *args, **kwargs)[source]

Create a Dataset from a csv file.

Parameters

  • filepath (str) – The file path or buffer to load from. Same as pandas.read_csv().

  • *args – Argument list for pandas.read_csv().

  • **kwargs – Keyword arguments for pandas.read_csv().

Returns

The constructed datapanel.

Return type

DataPanel

classmethod from_dict(d: Dict) → DataPanel[source]

Convert a dictionary to a dataset.

Alias for Dataset.from_batch(..).

classmethod from_feather(path: str)[source]

Create a Dataset from a feather file.

classmethod from_huggingface(*args, **kwargs)[source]

Load a Huggingface dataset as a DataPanel.

Use this to replace datasets.load_dataset, so

>>> dict_of_datasets = datasets.load_dataset('boolq')

becomes

>>> dict_of_datapanels = DataPanel.from_huggingface('boolq')
classmethod from_jsonl(json_path: str) → DataPanel[source]

Load a dataset from a .jsonl file on disk, where each line of the json file consists of a single example.

classmethod from_pandas(df: DataFrame)[source]

Create a Dataset from a pandas DataFrame.

groupby(*args, **kwargs)[source]
head(n: int = 5) → DataPanel[source]

Get the first n examples of the DataPanel.

items()[source]
keys()[source]
map(function: Optional[Callable] = None, with_indices: bool = False, input_columns: Optional[Union[str, List[str]]] = None, is_batched_fn: bool = False, batch_size: Optional[int] = 1, drop_last_batch: bool = False, num_workers: int = 0, output_type: Union[type, Dict[str, type]] = None, mmap: bool = False, mmap_path: str = None, materialize: bool = True, pbar: bool = False, **kwargs) → Optional[Union[Dict, List, AbstractColumn]][source]
mean(*args, **kwargs) → DataPanel[source]
merge(right: DataPanel, how: str = 'inner', on: Optional[Union[str, List[str]]] = None, left_on: Optional[Union[str, List[str]]] = None, right_on: Optional[Union[str, List[str]]] = None, sort: bool = False, suffixes: Sequence[str] = ('_x', '_y'), validate=None)[source]
classmethod read(path: str, *args, **kwargs) → DataPanel[source]

Load a DataPanel stored on disk.

remove_column(column: str) → None[source]

Remove a column from the dataset.

sample(n: Optional[int] = None, frac: Optional[float] = None, replace: bool = False, weights: Optional[Union[str, ndarray]] = None, random_state: Optional[Union[int, RandomState]] = None) → DataPanel[source]

Select a random sample of rows from DataPanel. Roughly equivalent to sample in Pandas https://pandas.pydata.org/docs/reference/api/panda s.DataFrame.sample.html.

Parameters

  • n (int) – Number of samples to draw. If frac is specified, this parameter should not be passed. Defaults to 1 if frac is not passed.

  • frac (float) – Fraction of rows to sample. If n is specified, this parameter should not be passed.

  • replace (bool) – Sample with or without replacement. Defaults to False.

  • weights (Union[str, np.ndarray]) – Weights to use for sampling. If None (default), the rows will be sampled uniformly. If a numpy array, the sample will be weighted accordingly. If a string, the weights will be applied to the rows based on the column with the name specified. If weights do not sum to 1 they will be normalized to sum to 1.

  • random_state (Union[int, np.random.RandomState]) – Random state or seed to use for sampling.

Returns

A random sample of rows from the DataPanel.

Return type

DataPanel

sort(by: Union[str, List[str]], ascending: Union[bool, List[bool]] = True, kind: str = 'quicksort') → DataPanel[source]

Sort the DataPanel by the values in the specified columns. Similar to sort_values in pandas.

Parameters

  • by (Union[str, List[str]]) – The columns to sort by.

  • ascending (Union[bool, List[bool]]) – Whether to sort in ascending or descending order. If a list, must be the same length as by.Defaults to True.

  • kind (str) – The kind of sort to use. Defaults to ‘quicksort’. Options include ‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’.

Returns

A sorted view of DataPanel.

Return type

DataPanel

streamlit()[source]
tail(n: int = 5) → DataPanel[source]

Get the last n examples of the DataPanel.

to_jsonl(path: str) → None[source]

Save a Dataset to a jsonl file.

to_pandas() → DataFrame[source]

Convert a Dataset to a pandas DataFrame.

update(function: Optional[Callable] = None, with_indices: bool = False, input_columns: Optional[Union[str, List[str]]] = None, is_batched_fn: bool = False, batch_size: Optional[int] = 1, remove_columns: Optional[List[str]] = None, num_workers: int = 0, output_type: Union[type, Dict[str, type]] = None, mmap: bool = False, mmap_path: str = None, materialize: bool = True, pbar: bool = False, **kwargs) → DataPanel[source]

Update the columns of the dataset.

values()[source]
write(path: str) → None[source]

Save a DataPanel to disk.

property columns

Column names in the DataPanel.

property data: BlockManager

Get the underlying data (excluding invisible rows).

To access underlying data with invisible rows, use _data.

logdir: Path = PosixPath('/home/docs/meerkat')
property ncols

Number of rows in the DataPanel.

property nrows

Number of rows in the DataPanel.

property shape

Shape of the DataPanel (num_rows, num_columns).

meerkat.display module

audio_file_formatter(cell: FileCell) → str[source]
auto_formatter(cell: Any)[source]
image_file_formatter(cell: FileCell)[source]
image_formatter(cell: Image)[source]
lambda_cell_formatter(cell: LambdaCell)[source]

meerkat.errors module

exception ConcatError[source]

Bases: ValueError

exception ConcatWarning[source]

Bases: RuntimeWarning

exception ConsolidationError[source]

Bases: ValueError

exception ExperimentalWarning[source]

Bases: FutureWarning

exception ImmutableError[source]

Bases: ValueError

exception MergeError[source]

Bases: ValueError

meerkat.provenance module

class ProvenanceMixin(*args, **kwargs)[source]

Bases: object

get_provenance(include_columns: bool = False, last_parent_only: bool = False)[source]
property node
class ProvenanceNode[source]

Bases: object

add_child(node: ProvenanceNode, key: Tuple)[source]
add_parent(node: ProvenanceNode, key: Tuple)[source]
cache_repr()[source]
get_provenance(last_parent_only: bool = False)[source]
property children
property last_parent
property parents
class ProvenanceObjNode(obj: ProvenanceMixin)[source]

Bases: ProvenanceNode

class ProvenanceOpNode(fn: callable, inputs: dict, outputs: object, captured_args: dict)[source]

Bases: ProvenanceNode

class provenance(enabled: bool = True)[source]

Bases: object

capture_provenance(capture_args: Optional[Sequence[str]] = None)[source]
get_nested_objs(data)[source]

Recursively get DataPanels and Columns from nested collections.

is_provenance_enabled()[source]
set_provenance(enabled=True)[source]
visualize_provenance(obj: Union[ProvenanceObjNode, ProvenanceOpNode], show_columns: bool = False, last_parent_only: bool = False)[source]

meerkat.version module

Module contents

Meerkat.

class AbstractCell(*args, **kwargs)[source]

Bases: ABC

get(*args, **kwargs) → object[source]

Get me the thing that this cell exists for.

loader(*args, **kwargs) → object[source]
property metadata: dict

Get the metadata associated with this cell.

class AbstractColumn(data: Optional[Sequence] = None, collate_fn: Optional[Callable] = None, formatter: Optional[Callable] = None, *args, **kwargs)[source]

Bases: BlockableMixin, CloneableMixin, CollateMixin, ColumnIOMixin, FunctionInspectorMixin, LambdaMixin, MappableMixin, MaterializationMixin, ProvenanceMixin, ABC

An abstract class for Meerkat columns.

append(column: AbstractColumn) → None[source]
argsort(ascending: Union[bool, List[bool]] = True, kind: str = 'quicksort') → AbstractColumn[source]

Return indices that would sorted the column.

Parameters

  • ascending (Union[bool, List[bool]]) – Whether to sort in ascending or descending order. If a list, must be the same length as by. Defaults to True.

  • kind (str) – The kind of sort to use. Defaults to ‘quicksort’. Options include ‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’.

Returns

A view of the column with the sorted data.

Return type

AbstractColumn

batch(batch_size: int = 1, drop_last_batch: bool = False, collate: bool = True, num_workers: int = 0, materialize: bool = True, *args, **kwargs)[source]

Batch the column.

Parameters

  • batch_size – integer batch size

  • drop_last_batch – drop the last batch if its smaller than batch_size

  • collate – whether to collate the returned batches

Returns

batches of data

static concat(columns: Sequence[AbstractColumn]) → None[source]
filter(function: Callable, with_indices=False, input_columns: Optional[Union[str, List[str]]] = None, is_batched_fn: bool = False, batch_size: Optional[int] = 1, drop_last_batch: bool = False, num_workers: Optional[int] = 0, materialize: bool = True, pbar: bool = False, **kwargs) → Optional[AbstractColumn][source]

Filter the elements of the column using a function.

classmethod from_data(data: Union[Columnable, AbstractColumn])[source]

Convert data to a meerkat column using the appropriate Column type.

full_length()[source]
classmethod get_writer(mmap: bool = False, template: Optional[AbstractColumn] = None)[source]
head(n: int = 5) → AbstractColumn[source]

Get the first n examples of the column.

is_equal(other: AbstractColumn) → bool[source]

Tests whether two columns.

Parameters

other (AbstractColumn) – [description]

sample(n: Optional[int] = None, frac: Optional[float] = None, replace: bool = False, weights: Optional[Union[str, ndarray]] = None, random_state: Optional[Union[int, RandomState]] = None) → AbstractColumn[source]

Select a random sample of rows from Column. Roughly equivalent to sample in Pandas https://pandas.pydata.org/docs/reference/api/panda s.DataFrame.sample.html.

Parameters

  • n (int) – Number of samples to draw. If frac is specified, this parameter should not be passed. Defaults to 1 if frac is not passed.

  • frac (float) – Fraction of rows to sample. If n is specified, this parameter should not be passed.

  • replace (bool) – Sample with or without replacement. Defaults to False.

  • weights (np.ndarray) – Weights to use for sampling. If None (default), the rows will be sampled uniformly. If a numpy array, the sample will be weighted accordingly. If weights do not sum to 1 they will be normalized to sum to 1.

  • random_state (Union[int, np.random.RandomState]) – Random state or seed to use for sampling.

Returns

A random sample of rows from the DataPanel.

Return type

AbstractColumn

sort(ascending: Union[bool, List[bool]] = True, kind: str = 'quicksort') → AbstractColumn[source]

Return a sorted view of the column.

Parameters

  • ascending (Union[bool, List[bool]]) – Whether to sort in ascending or descending order. If a list, must be the same length as by. Defaults to True.

  • kind (str) – The kind of sort to use. Defaults to ‘quicksort’. Options include ‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’.

Returns

A view of the column with the sorted data.

Return type

AbstractColumn

streamlit()[source]
tail(n: int = 5) → AbstractColumn[source]

Get the last n examples of the column.

to_pandas() → Series[source]
Columnable

alias of Union[Sequence, ndarray, Series, Tensor]

property data

Get the underlying data.

property formatter: Callable
property is_mmap
logdir: Path = PosixPath('/home/docs/meerkat')
property metadata
class ArrowArrayColumn(data: Sequence, *args, **kwargs)[source]

Bases: AbstractColumn

block_class

alias of ArrowBlock

classmethod concat(columns: Sequence[ArrowArrayColumn])[source]
is_equal(other: AbstractColumn) → bool[source]

Tests whether two columns.

Parameters

other (AbstractColumn) – [description]

to_numpy()[source]
to_pandas()[source]
to_tensor()[source]
class AudioColumn(data: Optional[Sequence[str]] = None, loader: Optional[callable] = None, transform: Optional[callable] = None, base_dir: Optional[str] = None, *args, **kwargs)[source]

Bases: FileColumn

A lambda column where each cell represents an audio file on disk. The underlying data is a PandasSeriesColumn of strings, where each string is the path to an image. The column materializes the images into memory when indexed. If the column is lazy indexed with the lz indexer, the images are not materialized and an FileCell or an AudioColumn is returned instead.

Parameters

  • data (Sequence[str]) – A list of filepaths to images.

  • transform (callable) –

    A function that transforms the image (e.g. torchvision.transforms.functional.center_crop).

    Warning

    In order for the column to be serializable, the transform function must be pickleable.

  • loader (callable) –

    A callable with signature def loader(filepath: str) -> PIL.Image:. Defaults to torchvision.datasets.folder.default_loader.

    Warning

    In order for the column to be serializable with write(), the loader function must be pickleable.

  • base_dir (str) – A base directory that the paths in data are relative to. If None, the paths are assumed to be absolute.

collate(batch)[source]

Collate data.

classmethod default_loader(*args, **kwargs)[source]
class CellColumn(cells: Optional[Sequence[AbstractCell]] = None, *args, **kwargs)[source]

Bases: AbstractColumn

static concat(columns: Sequence[CellColumn])[source]
classmethod from_cells(cells: Sequence[AbstractCell], *args, **kwargs)[source]
is_equal(other: AbstractColumn) → bool[source]

Tests whether two columns.

Parameters

other (AbstractColumn) – [description]

property cells
class DataPanel(data: Optional[Union[dict, list]] = None, *args, **kwargs)[source]

Bases: CloneableMixin, FunctionInspectorMixin, LambdaMixin, MappableMixin, MaterializationMixin, ProvenanceMixin

Meerkat DataPanel class.

add_column(name: str, data: Union[Sequence, ndarray, Series, Tensor], overwrite=False) → None[source]

Add a column to the DataPanel.

append(dp: DataPanel, axis: Union[str, int] = 'rows', suffixes: Tuple[str] = None, overwrite: bool = False) → DataPanel[source]

Append a batch of data to the dataset.

example_or_batch must have the same columns as the dataset (regardless of what columns are visible).

batch(batch_size: int = 1, drop_last_batch: bool = False, num_workers: int = 0, materialize: bool = True, shuffle: bool = False, *args, **kwargs)[source]

Batch the dataset. TODO:

Parameters

  • batch_size – integer batch size

  • drop_last_batch – drop the last batch if its smaller than batch_size

Returns

batches of data

consolidate()[source]
filter(function: Optional[Callable] = None, with_indices=False, input_columns: Optional[Union[str, List[str]]] = None, is_batched_fn: bool = False, batch_size: Optional[int] = 1, drop_last_batch: bool = False, num_workers: int = 0, materialize: bool = True, pbar: bool = False, **kwargs) → Optional[DataPanel][source]

Filter operation on the DataPanel.

classmethod from_arrow(table: Table)[source]

Create a Dataset from a pandas DataFrame.

classmethod from_batch(batch: Dict[str, Union[List, AbstractColumn]]) → DataPanel[source]

Convert a batch to a Dataset.

classmethod from_batches(batches: Sequence[Dict[str, Union[List, AbstractColumn]]]) → DataPanel[source]

Convert a list of batches to a dataset.

classmethod from_csv(filepath: str, *args, **kwargs)[source]

Create a Dataset from a csv file.

Parameters

  • filepath (str) – The file path or buffer to load from. Same as pandas.read_csv().

  • *args – Argument list for pandas.read_csv().

  • **kwargs – Keyword arguments for pandas.read_csv().

Returns

The constructed datapanel.

Return type

DataPanel

classmethod from_dict(d: Dict) → DataPanel[source]

Convert a dictionary to a dataset.

Alias for Dataset.from_batch(..).

classmethod from_feather(path: str)[source]

Create a Dataset from a feather file.

classmethod from_huggingface(*args, **kwargs)[source]

Load a Huggingface dataset as a DataPanel.

Use this to replace datasets.load_dataset, so

>>> dict_of_datasets = datasets.load_dataset('boolq')

becomes

>>> dict_of_datapanels = DataPanel.from_huggingface('boolq')
classmethod from_jsonl(json_path: str) → DataPanel[source]

Load a dataset from a .jsonl file on disk, where each line of the json file consists of a single example.

classmethod from_pandas(df: DataFrame)[source]

Create a Dataset from a pandas DataFrame.

groupby(*args, **kwargs)[source]
head(n: int = 5) → DataPanel[source]

Get the first n examples of the DataPanel.

items()[source]
keys()[source]
map(function: Optional[Callable] = None, with_indices: bool = False, input_columns: Optional[Union[str, List[str]]] = None, is_batched_fn: bool = False, batch_size: Optional[int] = 1, drop_last_batch: bool = False, num_workers: int = 0, output_type: Union[type, Dict[str, type]] = None, mmap: bool = False, mmap_path: str = None, materialize: bool = True, pbar: bool = False, **kwargs) → Optional[Union[Dict, List, AbstractColumn]][source]
mean(*args, **kwargs) → DataPanel[source]
merge(right: DataPanel, how: str = 'inner', on: Optional[Union[str, List[str]]] = None, left_on: Optional[Union[str, List[str]]] = None, right_on: Optional[Union[str, List[str]]] = None, sort: bool = False, suffixes: Sequence[str] = ('_x', '_y'), validate=None)[source]
classmethod read(path: str, *args, **kwargs) → DataPanel[source]

Load a DataPanel stored on disk.

remove_column(column: str) → None[source]

Remove a column from the dataset.

sample(n: Optional[int] = None, frac: Optional[float] = None, replace: bool = False, weights: Optional[Union[str, ndarray]] = None, random_state: Optional[Union[int, RandomState]] = None) → DataPanel[source]

Select a random sample of rows from DataPanel. Roughly equivalent to sample in Pandas https://pandas.pydata.org/docs/reference/api/panda s.DataFrame.sample.html.

Parameters

  • n (int) – Number of samples to draw. If frac is specified, this parameter should not be passed. Defaults to 1 if frac is not passed.

  • frac (float) – Fraction of rows to sample. If n is specified, this parameter should not be passed.

  • replace (bool) – Sample with or without replacement. Defaults to False.

  • weights (Union[str, np.ndarray]) – Weights to use for sampling. If None (default), the rows will be sampled uniformly. If a numpy array, the sample will be weighted accordingly. If a string, the weights will be applied to the rows based on the column with the name specified. If weights do not sum to 1 they will be normalized to sum to 1.

  • random_state (Union[int, np.random.RandomState]) – Random state or seed to use for sampling.

Returns

A random sample of rows from the DataPanel.

Return type

DataPanel

sort(by: Union[str, List[str]], ascending: Union[bool, List[bool]] = True, kind: str = 'quicksort') → DataPanel[source]

Sort the DataPanel by the values in the specified columns. Similar to sort_values in pandas.

Parameters

  • by (Union[str, List[str]]) – The columns to sort by.

  • ascending (Union[bool, List[bool]]) – Whether to sort in ascending or descending order. If a list, must be the same length as by.Defaults to True.

  • kind (str) – The kind of sort to use. Defaults to ‘quicksort’. Options include ‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’.

Returns

A sorted view of DataPanel.

Return type

DataPanel

streamlit()[source]
tail(n: int = 5) → DataPanel[source]

Get the last n examples of the DataPanel.

to_jsonl(path: str) → None[source]

Save a Dataset to a jsonl file.

to_pandas() → DataFrame[source]

Convert a Dataset to a pandas DataFrame.

update(function: Optional[Callable] = None, with_indices: bool = False, input_columns: Optional[Union[str, List[str]]] = None, is_batched_fn: bool = False, batch_size: Optional[int] = 1, remove_columns: Optional[List[str]] = None, num_workers: int = 0, output_type: Union[type, Dict[str, type]] = None, mmap: bool = False, mmap_path: str = None, materialize: bool = True, pbar: bool = False, **kwargs) → DataPanel[source]

Update the columns of the dataset.

values()[source]
write(path: str) → None[source]

Save a DataPanel to disk.

property columns

Column names in the DataPanel.

property data: BlockManager

Get the underlying data (excluding invisible rows).

To access underlying data with invisible rows, use _data.

logdir: Path = PosixPath('/home/docs/meerkat')
property ncols

Number of rows in the DataPanel.

property nrows

Number of rows in the DataPanel.

property shape

Shape of the DataPanel (num_rows, num_columns).

class FileCell(data: LambdaCellOp)[source]

Bases: LambdaCell

from_filepath(transform: Optional[callable] = None, loader: Optional[callable] = None, path: Optional[str] = None, base_dir: Optional[str] = None)[source]
property absolute_path
class FileColumn(data: Optional[Sequence[str]] = None, loader: Optional[callable] = None, transform: Optional[callable] = None, base_dir: Optional[str] = None, *args, **kwargs)[source]

Bases: LambdaColumn

A column where each cell represents an file stored on disk or the web. The underlying data is a PandasSeriesColumn of strings, where each string is the path to a file. The column materializes the files into memory when indexed. If the column is lazy indexed with the lz indexer, the files are not materialized and a FileCell or a FileColumn is returned instead.

Parameters

  • data (Sequence[str]) – A list of filepaths to images.

  • transform (callable) –

    A function that transforms the image (e.g. torchvision.transforms.functional.center_crop).

    Warning

    In order for the column to be serializable, the transform function must be pickleable.

  • loader (callable) –

    A callable with signature def loader(filepath: str) -> PIL.Image:. Defaults to torchvision.datasets.folder.default_loader.

    Warning

    In order for the column to be serializable with write(), the loader function must be pickleable.

  • base_dir (str) – A base directory that the paths in data are relative to. If None, the paths are assumed to be absolute.

classmethod default_loader(*args, **kwargs)[source]
classmethod from_filepaths(filepaths: Sequence[str], loader: Optional[callable] = None, transform: Optional[callable] = None, base_dir: Optional[str] = None)[source]
is_equal(other: AbstractColumn) → bool[source]

Tests whether two columns.

Parameters

other (AbstractColumn) – [description]

property base_dir
property loader
property transform
class ImageColumn(data: Optional[Sequence[str]] = None, loader: Optional[callable] = None, transform: Optional[callable] = None, base_dir: Optional[str] = None, *args, **kwargs)[source]

Bases: FileColumn

A column where each cell represents an image stored on disk. The underlying data is a PandasSeriesColumn of strings, where each string is the path to an image. The column materializes the images into memory when indexed. If the column is lazy indexed with the lz indexer, the images are not materialized and an ImageCell or an ImageColumn is returned instead.

Parameters

  • data (Sequence[str]) – A list of filepaths to images.

  • transform (callable) –

    A function that transforms the image (e.g. torchvision.transforms.functional.center_crop).

    Warning

    In order for the column to be serializable, the transform function must be pickleable.

  • loader (callable) –

    A callable with signature def loader(filepath: str) -> PIL.Image:. Defaults to torchvision.datasets.folder.default_loader.

    Warning

    In order for the column to be serializable with write(), the loader function must be pickleable.

  • base_dir (str) – A base directory that the paths in data are relative to. If None, the paths are assumed to be absolute.

classmethod default_loader(*args, **kwargs)[source]
class LambdaCell(data: LambdaCellOp)[source]

Bases: AbstractCell

get(*args, **kwargs)[source]

Get me the thing that this cell exists for.

property data: object

Get the data associated with this cell.

class LambdaColumn(data: Union[LambdaOp, BlockView], output_type: Optional[type] = None, *args, **kwargs)[source]

Bases: AbstractColumn

block_class

alias of LambdaBlock

static concat(columns: Sequence[LambdaColumn])[source]
is_equal(other: AbstractColumn) → bool[source]

Tests whether two columns.

Parameters

other (AbstractColumn) – [description]

property fn: Callable

Subclasses like ImageColumn should be able to implement their own version.

class ListColumn(data: Optional[Sequence] = None, *args, **kwargs)[source]

Bases: AbstractColumn

batch(batch_size: int = 1, drop_last_batch: bool = False, collate: bool = True, *args, **kwargs)[source]

Batch the column.

Parameters

  • batch_size – integer batch size

  • drop_last_batch – drop the last batch if its smaller than batch_size

  • collate – whether to collate the returned batches

Returns

batches of data

classmethod concat(columns: Sequence[ListColumn])[source]
default_formatter()
classmethod from_list(data: Sequence)[source]
is_equal(other: AbstractColumn) → bool[source]

Tests whether two columns.

Parameters

other (AbstractColumn) – [description]

class MedicalVolumeCell(paths: Union[str, Path, PathLike, Sequence[Union[str, Path, PathLike]]], loader: Optional[Callable] = None, transform: Optional[Callable] = None, cache_metadata: bool = False, *args, **kwargs)[source]

Bases: PathsMixin, AbstractCell

Interface for loading medical volume data.

Examples

# Specify xray dicoms with default orientation ("SI", "AP"): >>> cell = MedicalVolumeCell(“/path/to/xray.dcm”, loader=DicomReader(group_by=None, default_ornt=(“SI”, “AP”))

# Load multi-echo MRI volumes >>> cell = MedicalVolumeCell(“/path/to/mri/scan/dir”, loader=DicomReader(group_by=”EchoNumbers”))

clear_metadata()[source]
classmethod default_loader(paths: Sequence[Path], *args, **kwargs)[source]
classmethod from_state(state, *args, **kwargs)[source]
get(*args, cache_metadata: Optional[bool] = None, **kwargs)[source]

Get me the thing that this cell exists for.

get_metadata(ignore_bytes: bool = False, readable: bool = False, as_raw_type: bool = False, force_load: bool = False) → Dict[source]
get_state()[source]
class MedicalVolumeColumn(*args, **kwargs)[source]

Bases: CellColumn

classmethod from_filepaths(filepaths: Optional[Sequence[str]] = None, loader: Optional[callable] = None, transform: Optional[callable] = None, *args, **kwargs)[source]
class NumpyArrayColumn(data: Sequence, *args, **kwargs)[source]

Bases: AbstractColumn, NDArrayOperatorsMixin

block_class

alias of NumpyBlock

argsort(ascending: Union[bool, List[bool]] = True, kind: str = 'quicksort') → NumpyArrayColumn[source]

Return indices that would sorted the column.

Parameters

  • ascending (Union[bool, List[bool]]) – Whether to sort in ascending or descending order. If a list, must be the same length as by. Defaults to True.

  • kind (str) – The kind of sort to use. Defaults to ‘quicksort’. Options include ‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’.

Returns

A view of the column with the sorted data.

Return type

NumpySeriesColumn

For now! Raises error when shape of input array is more than one error.

classmethod concat(columns: Sequence[NumpyArrayColumn])[source]
classmethod from_array(data: ndarray, *args, **kwargs)[source]
classmethod from_npy(path, mmap_mode=None, allow_pickle=False, fix_imports=True, encoding='ASCII')[source]
classmethod get_writer(mmap: bool = False, template: Optional[AbstractColumn] = None)[source]
is_equal(other: AbstractColumn) → bool[source]

Tests whether two columns.

Parameters

other (AbstractColumn) – [description]

sort(ascending: Union[bool, List[bool]] = True, axis: int = -1, kind: str = 'quicksort', order: Optional[Union[str, List[str]]] = None) → NumpyArrayColumn[source]

Return a sorted view of the column.

Parameters

  • ascending (Union[bool, List[bool]]) – Whether to sort in ascending or descending order. If a list, must be the same length as by. Defaults to True.

  • kind (str) – The kind of sort to use. Defaults to ‘quicksort’. Options include ‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’.

Returns

A view of the column with the sorted data.

Return type

AbstractColumn

to_numpy() → ndarray[source]
to_pandas() → Series[source]
to_tensor() → Tensor[source]

Use column.to_tensor() instead of torch.tensor(column), which is very slow.

property is_mmap
class PandasSeriesColumn(data: Optional[Sequence] = None, collate_fn: Optional[Callable] = None, formatter: Optional[Callable] = None, *args, **kwargs)[source]

Bases: AbstractColumn, NDArrayOperatorsMixin

block_class

alias of PandasBlock

cat

alias of _MeerkatCategoricalAccessor

dt

alias of _MeerkatCombinedDatetimelikeProperties

str

alias of _MeerkatStringMethods

argsort(ascending: bool = True, kind: str = 'quicksort') → PandasSeriesColumn[source]

Return indices that would sorted the column.

Parameters

  • ascending (Union[bool, List[bool]]) – Whether to sort in ascending or descending order. If a list, must be the same length as by. Defaults to True.

  • kind (str) – The kind of sort to use. Defaults to ‘quicksort’. Options include ‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’.

Returns

A view of the column with the sorted data.

Return type

PandasSeriesColumn

For now! Raises error when shape of input array is more than one error.

classmethod concat(columns: Sequence[PandasSeriesColumn])[source]
classmethod from_array(data: ndarray, *args, **kwargs)[source]
is_equal(other: AbstractColumn) → bool[source]

Tests whether two columns.

Parameters

other (AbstractColumn) – [description]

sort(ascending: Union[bool, List[bool]] = True, kind: str = 'quicksort') → PandasSeriesColumn[source]

Return a sorted view of the column.

Parameters

  • ascending (Union[bool, List[bool]]) – Whether to sort in ascending or descending order. If a list, must be the same length as by. Defaults to True.

  • kind (str) – The kind of sort to use. Defaults to ‘quicksort’. Options include ‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’.

Returns

A view of the column with the sorted data.

Return type

AbstractColumn

to_numpy() → Tensor[source]
to_pandas() → Series[source]
to_tensor() → Tensor[source]

Use column.to_tensor() instead of torch.tensor(column), which is very slow.

class SpacyColumn(data: Sequence[spacy_tokens.Doc] = None, *args, **kwargs)[source]

Bases: ListColumn

classmethod from_docs(data: Sequence[spacy_tokens.Doc], *args, **kwargs)[source]
classmethod from_texts(texts: Sequence[str], lang: str = 'en_core_web_sm', *args, **kwargs)[source]
classmethod read(path: str, nlp: spacy.language.Language = None, lang: str = None, *args, **kwargs) → SpacyColumn[source]
write(path: str, **kwargs) → None[source]
property docs
property tokens
class TensorColumn(data: Optional[Sequence] = None, *args, **kwargs)[source]

Bases: NDArrayOperatorsMixin, AbstractColumn

block_class

alias of TensorBlock

argsort(ascending: Union[bool, List[bool]] = True, kind: str = 'quicksort') → TensorColumn[source]

Return indices that would sorted the column.

Parameters

  • ascending (Union[bool, List[bool]]) – Whether to sort in ascending or descending order. If a list, must be the same length as by. Defaults to True.

  • kind (str) – The kind of sort to use. Defaults to ‘quicksort’. Options include ‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’.

Returns

A view of the column with the sorted data.

Return type

TensorColumn

For now! Raises error when shape of input array is more than one error.

classmethod concat(columns: Sequence[TensorColumn])[source]
classmethod from_data(data: Union[Sequence, ndarray, Series, Tensor, AbstractColumn])[source]

Convert data to an EmbeddingColumn.

classmethod get_writer(mmap: bool = False, template: Optional[AbstractColumn] = None)[source]
is_equal(other: AbstractColumn) → bool[source]

Tests whether two columns.

Parameters

other (AbstractColumn) – [description]

sort(ascending: Union[bool, List[bool]] = True, kind: str = 'quicksort') → TensorColumn[source]

Return a sorted view of the column.

Parameters

  • ascending (Union[bool, List[bool]]) – Whether to sort in ascending or descending order. If a list, must be the same length as by. Defaults to True.

  • kind (str) – The kind of sort to use. Defaults to ‘quicksort’. Options include ‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’.

Returns

A view of the column with the sorted data.

Return type

AbstractColumn

to_numpy() → Series[source]
to_pandas() → Series[source]
to_tensor() → Tensor[source]
class provenance(enabled: bool = True)[source]

Bases: object

concat(objs: Union[Sequence[DataPanel], Sequence[AbstractColumn]], axis: Union[str, int] = 'rows', suffixes: Tuple[str] = None, overwrite: bool = False) → Union[DataPanel, AbstractColumn][source]

Concatenate a sequence of columns or a sequence of DataPanel`s. If sequence is empty, returns an empty `DataPanel.

  • If concatenating columns, all columns must be of the same type (e.g. all

ListColumn). - If concatenating `DataPanel`s along axis 0 (rows), all `DataPanel`s must have the same set of columns. - If concatenating `DataPanel`s along axis 1 (columns), all `DataPanel`s must have the same length and cannot have any of the same column names.

Parameters

  • objs (Union[Sequence[DataPanel], Sequence[AbstractColumn]]) – sequence of columns or DataPanels.

  • axis (Union[str, int]) – The axis along which to concatenate. Ignored if concatenating columns.

Returns

concatenated DataPanel or column

Return type

Union[DataPanel, AbstractColumn]

embed(data: DataPanel, input: str, encoder: Union[str, Encoder] = 'clip', modality: Optional[str] = None, out_col: Optional[str] = None, device: Union[int, str] = 'cpu', mmap_dir: Optional[str] = None, num_workers: int = 4, batch_size: int = 128, **kwargs) → DataPanel[source]

Embed a column of data with an encoder from the encoder registry.

Examples

Suppose you have an Image dataset (e.g. Imagenette, CIFAR-10) loaded into a Meerkat DataPanel. You can embed the images in the dataset with CLIP using a code snippet like:

import meerkat as mk

dp = mk.datasets.get("imagenette")

dp = mk.embed(
    data=dp,
    input_col="img",
    encoder="clip"
)
Parameters

  • data (mk.DataPanel) – A DataPanel containing the data to embed.

  • input_col (str) – The name of the column to embed.

  • encoder (Union[str, Encoder], optional) – Name of the encoder to use. List supported encoders with domino.encoders. Defaults to “clip”. Alternatively, pass an Encoder object containing a custom encoder.

  • modality (str, optional) – The modality of the data to be embedded. Defaults to None, in which case the modality is inferred from the type of the input column.

  • out_col (str, optional) – The name of the column where the embeddings are stored. Defaults to None, in which case it is "{encoder}({input_col})".

  • device (Union[int, str], optional) – The device on which. Defaults to “cpu”.

  • mmap_dir (str, optional) – The path to directory where a memory-mapped file containing the embeddings will be written. Defaults to None, in which case the embeddings are not memmapped.

  • num_workers (int, optional) – Number of worker processes used to load the data from disk. Defaults to 4.

  • batch_size (int, optional) – Size of the batches to used . Defaults to 128.

  • **kwargs – Additional keyword arguments are passed to the encoder. To see supported arguments for each encoder, see the encoder documentation (e.g. clip()).

Returns

A view of data with a new column containing the embeddings. This column will be named according to the out_col parameter.

Return type

mk.DataPanel

get(name: str, dataset_dir: Optional[str] = None, version: Optional[str] = None, download_mode: str = 'reuse', registry: Optional[str] = None, **kwargs) → Union[DataPanel, Dict[str, DataPanel]][source]

Load a dataset into .

Parameters

  • name (str) – Name of the dataset.

  • dataset_dir (str) – The directory containing dataset data. Defaults to ~/.meerkat/datasets/{name}.

  • version (str) – The version of the dataset. Defaults to latest.

  • download_mode (str) – The download mode. Options are: “reuse” (default) will download the dataset if it does not exist, “force” will download the dataset even if it exists, “extract” will reuse any downloaded archives but force extracting those archives, and “skip” will not download the dataset if it doesn’t yet exist. Defaults to reuse.

  • **kwargs – Additional arguments passed to the dataset.

merge(left: DataPanel, right: DataPanel, how: str = 'inner', on: Union[str, List[str]] = None, left_on: Union[str, List[str]] = None, right_on: Union[str, List[str]] = None, sort: bool = False, suffixes: Sequence[str] = ('_x', '_y'), validate=None)[source]
sample(data: Union[DataPanel, AbstractColumn], n: Optional[int] = None, frac: Optional[float] = None, replace: bool = False, weights: Optional[Union[str, ndarray]] = None, random_state: Optional[Union[int, RandomState]] = None) → Union[DataPanel, AbstractColumn][source]

Select a random sample of rows from DataPanel or Column. Roughly equivalent to sample in Pandas https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.sample.html.

Parameters

  • data (Union[DataPanel, AbstractColumn]) – DataPanel or Column to sample from.

  • n (int) – Number of samples to draw. If frac is specified, this parameter should not be passed. Defaults to 1 if frac is not passed.

  • frac (float) – Fraction of rows to sample. If n is specified, this parameter should not be passed.

  • replace (bool) – Sample with or without replacement. Defaults to False.

  • weights (Union[str, np.ndarray]) – Weights to use for sampling. If None (default), the rows will be sampled uniformly. If a numpy array, the sample will be weighted accordingly. If a string and data is a DataPanel, the weights will be applied to the rows based on the column with the name specified. If weights do not sum to 1 they will be normalized to sum to 1.

  • random_state (Union[int, np.random.RandomState]) – Random state or seed to use for sampling.

Returns

A random sample of rows from DataPanel or

Column.

Return type

Union[DataPanel, AbstractColumn]

sort(data: DataPanel, by: Union[str, List[str]], ascending: Union[bool, List[bool]] = True, kind: str = 'quicksort') → DataPanel[source]

Sort a DataPanel or Column. If a DataPanel, sort by the values in the specified columns. Similar to sort_values in pandas.

Parameters

  • data (Union[DataPanel, AbstractColumn]) – DataPanel or Column to sort.

  • by (Union[str, List[str]]) – The columns to sort by. Ignored if data is a Column.

  • ascending (Union[bool, List[bool]]) – Whether to sort in ascending or descending order. If a list, must be the same length as by.Defaults to True.

  • kind (str) – The kind of sort to use. Defaults to ‘quicksort’. Options include ‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’.

Returns

A sorted view of DataPanel.

Return type

DataPanel