Как лечить? 2019.03.05 01:22:40.535852 [ 6 ] {} <Trace> DB_NAME.TABLE_NAME

(StorageReplicatedMergeTree): Executing log entry to mutate part 20180411_20180411_0_0_0_8 to 20180411_20180411_0_0_0_9
2019.03.05 01:22:40.536493 [ 6 ] {} <Debug> DB_NAME.TABLE_NAME (SelectExecutor): Key condition: unknown
2019.03.05 01:22:40.536527 [ 6 ] {} <Debug> DB_NAME.TABLE_NAME (SelectExecutor): MinMax index condition: unknown
2019.03.05 01:22:40.536542 [ 6 ] {} <Debug> DB_NAME.TABLE_NAME (SelectExecutor): Selected 0 parts by date, 0 parts by key, 0 marks to read from 0 ranges
2019.03.05 01:22:40.536572 [ 6 ] {} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
2019.03.05 01:22:40.536641 [ 6 ] {} <Trace> Aggregator: Aggregating
2019.03.05 01:22:40.536669 [ 6 ] {} <Trace> Aggregator: Aggregation method: without_key
2019.03.05 01:22:40.536697 [ 6 ] {} <Trace> Aggregator: Aggregated. 0 to 1 rows (from 0.000 MiB) in 0.000 sec. (0.000 rows/sec., 0.000 MiB/sec.)
2019.03.05 01:22:40.536713 [ 6 ] {} <Trace> Aggregator: Merging aggregated data
2019.03.05 01:22:40.536747 [ 6 ] {} <Trace> virtual DB::MergingAndConvertingBlockInputStream::~MergingAndConvertingBlockInputStream(): Waiting for threads to finish
2019.03.05 01:22:40.536789 [ 6 ] {} <Trace> DB_NAME.TABLE_NAME (MergerMutator): Part 20180411_20180411_0_0_0_8 doesn't change up to mutation version 9
2019.03.05 01:22:40.536839 [ 6 ] {} <Debug> DB_NAME.TABLE_NAME (Data): Cloning part /database/clickhouse/data/DB_NAME/TABLE_NAME/20180411_20180411_0_0_0_8/ to /database/clickhouse/data/DB_NAME/TABLE_NAME/tmp_clone_20180411_20180411_0_0_0_9


2019.03.05 01:21:56.712125 [ 7 ] {} <Error> DB_NAME.TABLE_NAME (StorageReplicatedMergeTree): DB::StorageReplicatedMergeTree::queueTask()::<lambda(DB::StorageReplicatedMergeTree::LogEntryPtr&)>: Code: 107, e.displayText() = DB::Exception: /database/clickhouse/data/DB_NAME/TABLE_NAME/tmp_clone_20181112_20181130_0_16_1_25/duration_ms.bin doesn't exist, e.what() = DB::Exception, Stack trace:


0. clickhouse-server(StackTrace::StackTrace()+0x16) [0x568fcb6]
1. clickhouse-server(DB::Exception::Exception(std::string const&, int)+0x1f) [0x29b94cf]
2. clickhouse-server(DB::MergeTreeDataPartChecksum::checkSize(std::string const&) const+0x86) [0x51c72d6]
3. clickhouse-server(DB::MergeTreeDataPartChecksums::checkSizes(std::string const&) const+0x51) [0x51c7581]
4. clickhouse-server(DB::MergeTreeDataPart::checkConsistency(bool)+0x1d4) [0x51bcc24]
5. clickhouse-server(DB::MergeTreeData::cloneAndLoadDataPart(std::shared_ptr<DB::MergeTreeDataPart const> const&, std::string const&, DB::MergeTreePartInfo const&)+0x47b) [0x519930b]
6. clickhouse-server(DB::MergeTreeDataMergerMutator::mutatePartToTemporaryPart(DB::MergeTreeDataMergerMutator::FuturePart const&, std::vector<DB::MutationCommand, std::allocator<DB::MutationCommand> > const&, DB::Context const&)+0x651) [0x51b0181]
7. clickhouse-server(DB::StorageReplicatedMergeTree::tryExecutePartMutation(DB::ReplicatedMergeTreeLogEntry const&)+0x4ab) [0x512d32b]
8. clickhouse-server(DB::StorageReplicatedMergeTree::executeLogEntry(DB::ReplicatedMergeTreeLogEntry&)+0x89) [0x512e1d9]
9. clickhouse-server() [0x512eb50]
10. clickhouse-server(DB::ReplicatedMergeTreeQueue::processEntry(std::function<std::shared_ptr<zkutil::ZooKeeper> ()>, std::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&, std::function<bool (std::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&)>)+0x41) [0x521faa1]
11. clickhouse-server(DB::StorageReplicatedMergeTree::queueTask()+0x19a) [0x510572a]
12. clickhouse-server(DB::BackgroundProcessingPool::threadFunction()+0x396) [0x5162456]
13. clickhouse-server() [0x5d9675f]
14. /lib64/libpthread.so.0(+0x7e25) [0x7f55003dbe25]
15. /lib64/libc.so.6(clone+0x6d) [0x7f54ffbff34d] По этой таблице есть активные мутации которые не не могут исполнится. в моей сборке KILL mutation не работает
KILL MUTATION WHERE database = 'default' AND table = 'table'
;

Expected QUERY

короче: хотел сделать TRUNCATE этой таблицы. ( В моем случае это ок думал после транкейта все партиции удалятся). После этойго в process list. стали появляться запросы типа SELECT DISTINCT arrayJoin(extractAll(name, '[\\w_]{2,}')) AS res FROM (SELECT name FROM system.functions UNION ALL SELECT name FROM system.table_engines UNION ALL SELECT name FROM system.formats UNION ALL SELECT name FROM system.table_functions UNION ALL SELECT name FROM system.data_type_families UNION ALL SELECT name FROM system.settings UNION ALL SELECT concat(func.name, comb.name) FROM system.functions AS func CROSS JOIN system.aggregate_function_combinators AS comb WHERE is_aggregate UNION ALL SELECT name FROM system.databases LIMIT 10000 UNION ALL SELECT DISTINCT name FROM system.tables LIMIT 10000 UNION ALL SELECT DISTINCT name FROM system.columns LIMIT 10000) WHERE notEmpty(res), KILL query не помогал. Судя по всему после таймаута CH запускал такой же запрос опять. Принял решение рестартить ноду. Рестарт тоже не прошлел гладко. Сервер не мог корректно завершить работу, висело на собщении “Stopped being leader” как раз бля этой таблицы. Сделал KILL -9 Received termination signal (Terminated) и далее при старте CH визимо заигнорил битые партиции. Именно для этой таблицы я готов мириться с потерей даннхых, но вообще беда =( Эта таблица у меня реплицируемая, но в шарде всего одна репоика. Сделал это на будещее (Может в этом причина?)