Migrating from PyTables 2.x to 3.x¶
| Author: | Antonio Valentino |
|---|---|
| Author: | Anthony Scopatz |
| Author: | Thomas Provoost |
This document describes the major changes in PyTables in going from the 2.x to 3.x series and what you need to know when migrating downstream code bases.
Python 3 at Last!¶
The PyTables 3.x series now ships with full compatibility for Python 3.1+. Additionally, we plan on maintaining compatibility with Python 2.7 for the foreseeable future. Python 2.6 is no longer supported but may work in most cases. Note that the entire 3.x series now relies on numexpr v2.1+, which itself is the first version of numexpr support both Python 2 & 3.
Numeric, Numarray, NetCDF3, & HDF5 1.6 No More!¶
PyTables no longer supports numeric and numarray. Please use numpy instead.
Additionally, the tables.netcdf3 module has been removed. Please refer
to the netcdf4-python project for further support. Lastly, the older
HDF5 1.6 API is no longer supported. Please upgrade to HDF5 1.8+.
Unicode all the strings!¶
In Python 3, all strings are natively in Unicode. This introduces some difficulties, as the native HDF5 string format is not Unicode-compatible. To minimize explicit conversion troubles when writing, especially when creating data sets from existing Python objects, string objects are implicitly cast to non-Unicode for HDF5 storage. To make you aware of this, a warning is raised when this happens.
This is certainly no true Unicode compatibility, but mainly for convenience
with the pure-Unicode Python 3 string type. Any string that is not castable
as ascii upon creation of your data set, will hence still raise an error.
For true Unicode support, look into the VLUnicodeAtom class.
Major API Changes¶
The PyTables developers, by popular demand, have taken this opportunity that a major version number upgrade affords to implement significant API changes. We have tried to do this in such a way that will not immediately break most existing code, though in some breakages may still occur.
PEP 8 Compliance¶
The PyTables 3.x series now follows PEP 8 coding standard. This makes using PyTables more idiomatic with surrounding Python code that also adheres to this standard. The primary way that the 2.x series was not PEP 8 compliant was with respect to variable naming conventions. Approximately 450 API variables were identified and updated for PyTables 3.x.
To ease migration, PyTables ships with a new pt2to3 command line tool.
This tool will run over a file and replace any instances of the old variable
names with the 3.x version of the name. This tool covers the overwhelming
majority of cases was used to transition the PyTables code base itself! However,
it may also accidentally also pick up variable names in 3rd party codes that
have exactly the same name as a PyTables’ variable. This is because pt2to3
was implemented using regular expressions rather than a fancier AST-based
method. By using regexes, pt2to3 works on Python and Cython code.
pt2to3 help:
usage: pt2to3 [-h] [-r] [-p] [-o OUTPUT] [-i] filename
PyTables 2.x -> 3.x API transition tool This tool displays to standard out, so
it is common to pipe this to another file: $ pt2to3 oldfile.py > newfile.py
positional arguments:
filename path to input file.
optional arguments:
-h, --help show this help message and exit
-r, --reverse reverts changes, going from 3.x -> 2.x.
-p, --no-ignore-previous
ignores previous_api() calls.
-o OUTPUT output file to write to.
-i, --inplace overwrites the file in-place.
Note that pt2to3 only works on a single file, not a a directory. However,
a simple BASH script may be written to run pt2to3 over an entire directory
and all sub-directories:
#!/bin/bash
for f in $(find .)
do
echo $f
pt2to3 $f > temp.txt
mv temp.txt $f
done
Note
pt2to3 uses the argparse module that is part of the
Python standard library since Python 2.7.
Users of Python 2.6 should install argparse separately
(e.g. via pip).
The old APIs and variable names will continue to be supported for the short term, where possible. (The major backwards incompatible changes come from the renaming of some function and method arguments and keyword arguments.) Using the 2.x APIs in the 3.x series, however, will issue warnings. The following is the release plan for the warning types:
- 3.0 - PendingDeprecationWarning
- 3.1 - DeprecationWarning
- >=3.2 - Remove warnings, previous_api(), and _past.py; keep pt2to3,
The current plan is to maintain the old APIs for at least 2 years, though this is subject to change.
Consistent create_xxx() Signatures¶
Also by popular demand, it is now possible to create all data sets (Array,
CArray, EArray, VLArray, and Table) from existing Python objects.
Constructors for these classes now accept either of the following keyword arguments:
- an
objto initialize with data - or both
atomandshapeto initialize an empty structure, if possible.
These keyword arguments are also now part of the function signature for the
corresponding create_xxx() methods on the File class. These would be called
as follows:
# All create methods will support the following
create_xxx(where, name, obj=obj)
# All non-variable length arrays support the following:
create_xxx(where, name, atom=atom, shape=shape)
Using obj or atom and shape are mutually exclusive. Previously only
Array could be created with an existing Python object using the object
keyword argument.
API Name Changes¶
The following tables shows the old 2.x names that have been update to their
new values in the new 3.x series. Please use the pt2to3 tool to convert
between these.
| 2.x Name | 3.x Name |
|---|---|
| AtomFromHDF5Type | atom_from_hdf5_type |
| AtomToHDF5Type | atom_to_hdf5_type |
| BoolTypeNextAfter | bool_type_next_after |
| HDF5ClassToString | hdf5_class_to_string |
| HDF5ToNPExtType | hdf5_to_np_ext_type |
| HDF5ToNPNestedType | hdf5_to_np_nested_type |
| IObuf | iobuf |
| IObufcpy | iobufcpy |
| IntTypeNextAfter | int_type_next_after |
| NPExtPrefixesToPTKinds | npext_prefixes_to_ptkinds |
| PTSpecialKinds | pt_special_kinds |
| PTTypeToHDF5 | pttype_to_hdf5 |
| StringNextAfter | string_next_after |
| __allowedInitKwArgs | __allowed_init_kwargs |
| __getRootGroup | __get_root_group |
| __next__inKernel | __next__inkernel |
| _actionLogName | _action_log_name |
| _actionLogParent | _action_log_parent |
| _actionLogPath | _action_log_path |
| _addRowsToIndex | _add_rows_to_index |
| _appendZeros | _append_zeros |
| _autoIndex | _autoindex |
| _byteShape | _byte_shape |
| _c_classId | _c_classid |
| _c_shadowNameRE | _c_shadow_name_re |
| _cacheDescriptionData | _cache_description_data |
| _checkAndSetPair | _check_and_set_pair |
| _checkAttributes | _check_attributes |
| _checkBase | _checkbase |
| _checkColumn | _check_column |
| _checkGroup | _check_group |
| _checkNotClosed | _check_not_closed |
| _checkOpen | _check_open |
| _checkShape | _check_shape |
| _checkShapeAppend | _check_shape_append |
| _checkUndoEnabled | _check_undo_enabled |
| _checkWritable | _check_writable |
| _check_sortby_CSI | _check_sortby_csi |
| _closeFile | _close_file |
| _codeToOp | _code_to_op |
| _column__createIndex | _column__create_index |
| _compileCondition | _compile_condition |
| _conditionCache | _condition_cache |
| _convertTime64 | _convert_time64 |
| _convertTime64_ | _convert_time64_ |
| _convertTypes | _convert_types |
| _createArray | _create_array |
| _createCArray | _create_carray |
| _createMark | _create_mark |
| _createPath | _create_path |
| _createTable | _create_table |
| _createTransaction | _create_transaction |
| _createTransactionGroup | _create_transaction_group |
| _disableIndexingInQueries | _disable_indexing_in_queries |
| _doReIndex | _do_reindex |
| _emptyArrayCache | _empty_array_cache |
| _enableIndexingInQueries | _enable_indexing_in_queries |
| _enabledIndexingInQueries | _enabled_indexing_in_queries |
| _exprvarsCache | _exprvars_cache |
| _f_copyChildren | _f_copy_children |
| _f_delAttr | _f_delattr |
| _f_getAttr | _f_getattr |
| _f_getChild | _f_get_child |
| _f_isVisible | _f_isvisible |
| _f_iterNodes | _f_iter_nodes |
| _f_listNodes | _f_list_nodes |
| _f_setAttr | _f_setattr |
| _f_walkGroups | _f_walk_groups |
| _f_walkNodes | _f_walknodes |
| _fancySelection | _fancy_selection |
| _fillCol | _fill_col |
| _flushBufferedRows | _flush_buffered_rows |
| _flushFile | _flush_file |
| _flushModRows | _flush_mod_rows |
| _g_addChildrenNames | _g_add_children_names |
| _g_checkGroup | _g_check_group |
| _g_checkHasChild | _g_check_has_child |
| _g_checkName | _g_check_name |
| _g_checkNotContains | _g_check_not_contains |
| _g_checkOpen | _g_check_open |
| _g_closeDescendents | _g_close_descendents |
| _g_closeGroup | _g_close_group |
| _g_copyAsChild | _g_copy_as_child |
| _g_copyChildren | _g_copy_children |
| _g_copyRows | _g_copy_rows |
| _g_copyRows_optim | _g_copy_rows_optim |
| _g_copyWithStats | _g_copy_with_stats |
| _g_createHardLink | _g_create_hard_link |
| _g_delAndLog | _g_del_and_log |
| _g_delLocation | _g_del_location |
| _g_flushGroup | _g_flush_group |
| _g_getAttr | _g_getattr |
| _g_getChildGroupClass | _g_get_child_group_class |
| _g_getChildLeafClass | _g_get_child_leaf_class |
| _g_getGChildAttr | _g_get_gchild_attr |
| _g_getLChildAttr | _g_get_lchild_attr |
| _g_getLinkClass | _g_get_link_class |
| _g_listAttr | _g_list_attr |
| _g_listGroup | _g_list_group |
| _g_loadChild | _g_load_child |
| _g_logAdd | _g_log_add |
| _g_logCreate | _g_log_create |
| _g_logMove | _g_log_move |
| _g_maybeRemove | _g_maybe_remove |
| _g_moveNode | _g_move_node |
| _g_postInitHook | _g_post_init_hook |
| _g_postReviveHook | _g_post_revive_hook |
| _g_preKillHook | _g_pre_kill_hook |
| _g_propIndexes | _g_prop_indexes |
| _g_readCoords | _g_read_coords |
| _g_readSelection | _g_read_selection |
| _g_readSlice | _g_read_slice |
| _g_readSortedSlice | _g_read_sorted_slice |
| _g_refNode | _g_refnode |
| _g_removeAndLog | _g_remove_and_log |
| _g_setAttr | _g_setattr |
| _g_setLocation | _g_set_location |
| _g_setNestedNamesDescr | _g_set_nested_names_descr |
| _g_setPathNames | _g_set_path_names |
| _g_unrefNode | _g_unrefnode |
| _g_updateDependent | _g_update_dependent |
| _g_updateLocation | _g_update_location |
| _g_updateNodeLocation | _g_update_node_location |
| _g_updateTableLocation | _g_update_table_location |
| _g_widthWarning | _g_width_warning |
| _g_writeCoords | _g_write_coords |
| _g_writeSelection | _g_write_selection |
| _g_writeSlice | _g_write_slice |
| _getColumnInstance | _get_column_instance |
| _getConditionKey | _get_condition_key |
| _getContainer | _get_container |
| _getEnumMap | _get_enum_map |
| _getFileId | _get_file_id |
| _getFinalAction | _get_final_action |
| _getInfo | _get_info |
| _getLinkClass | _get_link_class |
| _getMarkID | _get_mark_id |
| _getNode | _get_node |
| _getOrCreatePath | _get_or_create_path |
| _getTypeColNames | _get_type_col_names |
| _getUnsavedNrows | _get_unsaved_nrows |
| _getValueFromContainer | _get_value_from_container |
| _hiddenNameRE | _hidden_name_re |
| _hiddenPathRE | _hidden_path_re |
| _indexNameOf | _index_name_of |
| _indexNameOf_ | _index_name_of_ |
| _indexPathnameOf | _index_pathname_of |
| _indexPathnameOfColumn | _index_pathname_of_column |
| _indexPathnameOfColumn_ | _index_pathname_of_column_ |
| _indexPathnameOf_ | _index_pathname_of_ |
| _initLoop | _init_loop |
| _initSortedSlice | _init_sorted_slice |
| _isWritable | _iswritable |
| _is_CSI | _is_csi |
| _killNode | _killnode |
| _lineChunkSize | _line_chunksize |
| _lineSeparator | _line_separator |
| _markColumnsAsDirty | _mark_columns_as_dirty |
| _newBuffer | _new_buffer |
| _notReadableError | _not_readable_error |
| _npSizeType | _npsizetype |
| _nxTypeFromNPType | _nxtype_from_nptype |
| _opToCode | _op_to_code |
| _openArray | _open_array |
| _openUnImplemented | _open_unimplemented |
| _pointSelection | _point_selection |
| _processRange | _process_range |
| _processRangeRead | _process_range_read |
| _pythonIdRE | _python_id_re |
| _reIndex | _reindex |
| _readArray | _read_array |
| _readCoordinates | _read_coordinates |
| _readCoords | _read_coords |
| _readIndexSlice | _read_index_slice |
| _readSelection | _read_selection |
| _readSlice | _read_slice |
| _readSortedSlice | _read_sorted_slice |
| _refNode | _refnode |
| _requiredExprVars | _required_expr_vars |
| _reservedIdRE | _reserved_id_re |
| _reviveNode | _revivenode |
| _saveBufferedRows | _save_buffered_rows |
| _searchBin | _search_bin |
| _searchBinNA_b | _search_bin_na_b |
| _searchBinNA_d | _search_bin_na_d |
| _searchBinNA_e | _search_bin_na_e |
| _searchBinNA_f | _search_bin_na_f |
| _searchBinNA_g | _search_bin_na_g |
| _searchBinNA_i | _search_bin_na_i |
| _searchBinNA_ll | _search_bin_na_ll |
| _searchBinNA_s | _search_bin_na_s |
| _searchBinNA_ub | _search_bin_na_ub |
| _searchBinNA_ui | _search_bin_na_ui |
| _searchBinNA_ull | _search_bin_na_ull |
| _searchBinNA_us | _search_bin_na_us |
| _setAttributes | _set_attributes |
| _setColumnIndexing | _set_column_indexing |
| _shadowName | _shadow_name |
| _shadowParent | _shadow_parent |
| _shadowPath | _shadow_path |
| _sizeToShape | _size_to_shape |
| _tableColumnPathnameOfIndex | _table_column_pathname_of_index |
| _tableFile | _table_file |
| _tablePath | _table_path |
| _table__autoIndex | _table__autoindex |
| _table__getautoIndex | _table__getautoindex |
| _table__setautoIndex | _table__setautoindex |
| _table__whereIndexed | _table__where_indexed |
| _transGroupName | _trans_group_name |
| _transGroupParent | _trans_group_parent |
| _transGroupPath | _trans_group_path |
| _transName | _trans_name |
| _transParent | _trans_parent |
| _transPath | _trans_path |
| _transVersion | _trans_version |
| _unrefNode | _unrefnode |
| _updateNodeLocations | _update_node_locations |
| _useIndex | _use_index |
| _vShape | _vshape |
| _vType | _vtype |
| _v__nodeFile | _v__nodefile |
| _v__nodePath | _v__nodepath |
| _v_colObjects | _v_colobjects |
| _v_maxGroupWidth | _v_max_group_width |
| _v_maxTreeDepth | _v_maxtreedepth |
| _v_nestedDescr | _v_nested_descr |
| _v_nestedFormats | _v_nested_formats |
| _v_nestedNames | _v_nested_names |
| _v_objectID | _v_objectid |
| _whereCondition | _where_condition |
| _writeCoords | _write_coords |
| _writeSelection | _write_selection |
| _writeSlice | _write_slice |
| appendLastRow | append_last_row |
| attrFromShadow | attr_from_shadow |
| attrToShadow | attr_to_shadow |
| autoIndex | autoindex |
| bufcoordsData | bufcoords_data |
| calcChunksize | calc_chunksize |
| checkFileAccess | check_file_access |
| checkNameValidity | check_name_validity |
| childName | childname |
| chunkmapData | chunkmap_data |
| classIdDict | class_id_dict |
| className | classname |
| classNameDict | class_name_dict |
| containerRef | containerref |
| convertToNPAtom | convert_to_np_atom |
| convertToNPAtom2 | convert_to_np_atom2 |
| copyChildren | copy_children |
| copyClass | copyclass |
| copyFile | copy_file |
| copyLeaf | copy_leaf |
| copyNode | copy_node |
| copyNodeAttrs | copy_node_attrs |
| countLoggedInstances | count_logged_instances |
| createArray | create_array |
| createCArray | create_carray |
| createCSIndex | create_csindex |
| createEArray | create_earray |
| createExternalLink | create_external_link |
| createGroup | create_group |
| createHardLink | create_hard_link |
| createIndex | create_index |
| createIndexesDescr | create_indexes_descr |
| createIndexesTable | create_indexes_table |
| createNestedType | create_nested_type |
| createSoftLink | create_soft_link |
| createTable | create_table |
| createVLArray | create_vlarray |
| defaultAutoIndex | default_auto_index |
| defaultIndexFilters | default_index_filters |
| delAttr | del_attr |
| delAttrs | _del_attrs |
| delNodeAttr | del_node_attr |
| detectNumberOfCores | detect_number_of_cores |
| disableUndo | disable_undo |
| dumpGroup | dump_group |
| dumpLeaf | dump_leaf |
| dumpLoggedInstances | dump_logged_instances |
| enableUndo | enable_undo |
| enumFromHDF5 | enum_from_hdf5 |
| enumToHDF5 | enum_to_hdf5 |
| fetchLoggedInstances | fetch_logged_instances |
| flushRowsToIndex | flush_rows_to_index |
| getAttr | get_attr |
| getAttrs | _get_attrs |
| getClassByName | get_class_by_name |
| getColsInOrder | get_cols_in_order |
| getCurrentMark | get_current_mark |
| getEnum | get_enum |
| getFilters | get_filters |
| getHDF5Version | get_hdf5_version |
| getIndices | get_indices |
| getLRUbounds | get_lru_bounds |
| getLRUsorted | get_lru_sorted |
| getLookupRange | get_lookup_range |
| getNestedField | get_nested_field |
| getNestedFieldCache | get_nested_field_cache |
| getNestedType | get_nested_type |
| getNode | get_node |
| getNodeAttr | get_node_attr |
| getPyTablesVersion | get_pytables_version |
| getTypeEnum | get_type_enum |
| getWhereList | get_where_list |
| hdf5Extension | hdf5extension |
| hdf5Version | hdf5_version |
| indexChunk | indexchunk |
| indexValid | indexvalid |
| indexValidData | index_valid_data |
| indexValues | indexvalues |
| indexValuesData | index_values_data |
| indexesExtension | indexesextension |
| infType | inftype |
| infinityF | infinityf |
| infinityMap | infinitymap |
| initRead | initread |
| isHDF5File | is_hdf5_file |
| isPyTablesFile | is_pytables_file |
| isUndoEnabled | is_undo_enabled |
| isVisible | isvisible |
| isVisibleName | isvisiblename |
| isVisibleNode | is_visible_node |
| isVisiblePath | isvisiblepath |
| is_CSI | is_csi |
| iterNodes | iter_nodes |
| iterseqMaxElements | iterseq_max_elements |
| joinPath | join_path |
| joinPaths | join_paths |
| linkExtension | linkextension |
| listLoggedInstances | list_logged_instances |
| listNodes | list_nodes |
| loadEnum | load_enum |
| logInstanceCreation | log_instance_creation |
| lrucacheExtension | lrucacheextension |
| metaIsDescription | MetaIsDescription |
| modifyColumn | modify_column |
| modifyColumns | modify_columns |
| modifyCoordinates | modify_coordinates |
| modifyRows | modify_rows |
| moveFromShadow | move_from_shadow |
| moveNode | move_node |
| moveToShadow | move_to_shadow |
| newNode | new_node |
| newSet | newset |
| newdstGroup | newdst_group |
| objectID | object_id |
| oldPathname | oldpathname |
| openFile | open_file |
| openNode | open_node |
| parentNode | parentnode |
| parentPath | parentpath |
| reIndex | reindex |
| reIndexDirty | reindex_dirty |
| readCoordinates | read_coordinates |
| readIndices | read_indices |
| readSlice | read_slice |
| readSorted | read_sorted |
| readWhere | read_where |
| read_sliceLR | read_slice_lr |
| recreateIndexes | recreate_indexes |
| redoAddAttr | redo_add_attr |
| redoCreate | redo_create |
| redoDelAttr | redo_del_attr |
| redoMove | redo_move |
| redoRemove | redo_remove |
| removeIndex | remove_index |
| removeNode | remove_node |
| removeRows | remove_rows |
| renameNode | rename_node |
| rootUEP | root_uep |
| searchLastRow | search_last_row |
| setAttr | set_attr |
| setAttrs | _set_attrs |
| setBloscMaxThreads | set_blosc_max_threads |
| setInputsRange | set_inputs_range |
| setNodeAttr | set_node_attr |
| setOutput | set_output |
| setOutputRange | set_output_range |
| silenceHDF5Messages | silence_hdf5_messages |
| splitPath | split_path |
| tableExtension | tableextension |
| undoAddAttr | undo_add_attr |
| undoCreate | undo_create |
| undoDelAttr | undo_del_attr |
| undoMove | undo_move |
| undoRemove | undo_remove |
| utilsExtension | utilsextension |
| walkGroups | walk_groups |
| walkNodes | walk_nodes |
| whereAppend | append_where |
| whereCond | wherecond |
| whichClass | which_class |
| whichLibVersion | which_lib_version |
| willQueryUseIndexing | will_query_use_indexing |
Enjoy data!
—The PyTables Developers
