Skip to content

KYLIN-5748 Skip build flat table for index build job if all new created indexes can be built from existed parent index#2162

Open
zznlime wants to merge 3 commits intoapache:kylin5from
zznlime:kylin-5748
Open

KYLIN-5748 Skip build flat table for index build job if all new created indexes can be built from existed parent index#2162
zznlime wants to merge 3 commits intoapache:kylin5from
zznlime:kylin-5748

Conversation

@zznlime
Copy link
Contributor

@zznlime zznlime commented Dec 26, 2023

For an index build job, if all the new added indexes can be built from their existed parent index. No index was built from new created flat table.
For this case, there won't be any data inconsistency across indexes. And there is no need to build a new flat table to serve the stage of index build by layer.
So I think we can skip stage BUILD DICT and stage GENERATE FLAT TABLE for an index build job in condition that all new added indexes can be built from their existed parent index

@zznlime
Copy link
Contributor Author

zznlime commented Dec 26, 2023

@hit-lacus PTAL, Thanks

@Mrhs121
Copy link
Contributor

Mrhs121 commented Mar 26, 2024

Your proposal will be invalid in this use case: if you add a new bitmap type metric to the model, you can build the new index separately, and the task type is also INDEX_BUIDL. If you skip building the dictionary, an error will occur when generating encoding columns for the flat table later.

@pfzhan
Copy link
Contributor

pfzhan commented Sep 2, 2024

@zznlime Does this pr need to be merged into the kylin code? If necessary, please send pr to @liyang-kylin . I will resolve this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants