issue/961: fix metax init with preload

[Ceng23333]

#961

issue fixed

cpu

lscpu
Architecture:           aarch64
  CPU op-mode(s):       32-bit, 64-bit
  Byte Order:           Little Endian
CPU(s):                 128
  On-line CPU(s) list:  0-127
Vendor ID:              Phytium
  BIOS Vendor ID:       PHYTIUM LTD.
  Model name:           Phytium,S5000C/64
    BIOS Model name:    S5000C
    Model:              0
    Thread(s) per core: 1
    Core(s) per socket: 64
    Socket(s):          2
    Stepping:           0x0
    BogoMIPS:           2000.00
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop sha3 sm3 s
                        m4 asimddp sha512

gpu

型号：Mars X203
驱动：HPCC Version: 2.32.0.6

性能指标

9G7B

python examples/jiuge.py --metax --model_path=/data-aisoft/mechdancer/models/9G7B_MHA --max_new_tokens=1024
Namespace(cpu=False, nvidia=False, metax=True, moore=False, iluvatar=False, cambricon=False, model_path='/data-aisoft/mechdancer/models/9G7B_MHA', max_new_tokens=1024, backend='cpp', batch_size=1, prompt='How are you', tp=1, enable_paged_attn=False)
 load weights ......
Processing: model.safetensors
Processing files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:03<00:00,  3.36s/it]
 load weights over! 3363.4326457977295 ms

<|im_start|>user
How are you<|im_end|>
<|im_start|>assistant
=================== start generate ====================



 Generation completed in 3349.49 ms
 Batchsize=1  Per_Batch_Input_Len=13  Per_Batch_New_Tokens=41

 Prefill TTFT: 0.86ms  Throughput: 15.11tok/s

 Decode  Avg ITL: 62.22ms   Throughput: 16.07tok/s

Hello! I'm doing well, thank you for asking. How about you? Is there anything specific you'd like to talk about or any questions you have? I'm here to help!
total_time: 3406.82 ms

python examples/jiuge.py --metax --model_path=/data-aisoft/mechdancer/models/9G7B_MHA --max_new_tokens=1024 --tp=4
Namespace(cpu=False, nvidia=False, metax=True, moore=False, iluvatar=False, cambricon=False, model_path='/data-aisoft/mechdancer/models/9G7B_MHA', max_new_tokens=1024, backend='cpp', batch_size=1, prompt='How are you', tp=4, enable_paged_attn=False)
 load weights ......
Processing: model.safetensors
Processing files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:05<00:00,  5.40s/it]
 load weights over! 5415.926456451416 ms

<|im_start|>user
How are you<|im_end|>
<|im_start|>assistant
=================== start generate ====================



 Generation completed in 4345.9 ms
 Batchsize=1  Per_Batch_Input_Len=13  Per_Batch_New_Tokens=27

 Prefill TTFT: 1.83ms  Throughput: 7.1tok/s

 Decode  Avg ITL: 96.73ms   Throughput: 10.34tok/s

Hello! I'm doing well, thank you for asking. How about you? Is there anything I can help you with today?
total_time: 4424.83 ms

python test/bench/test_benchmark.py --metax /data-aisoft/mechdancer/models/9G7B_MHA --bench ceval --subject middle_school_mathematics --num_samples 100 --backend cpp --ndev 1
============================================================
OVERALL RESULTS
============================================================
============================================================
Overall 成绩: 61/100 = 61.00%
Total Latency: 439.86021568998694 seconds
Total Tokens Processed: 25809 tokens
Overall Throughput: 58.68 tokens/s

9G8B

python examples/jiuge.py --metax --model_path=/data-aisoft/zenghua/models/9g_8b_thinking_llama/ --max_new_tokens=1024 --tp=1
Namespace(cpu=False, nvidia=False, metax=True, moore=False, iluvatar=False, cambricon=False, model_path='/data-aisoft/zenghua/models/9g_8b_thinking_llama/', max_new_tokens=1024, backend='cpp', batch_size=1, prompt='How are you', tp=1, enable_paged_attn=False)
 load weights ......
Processing: model-00001-of-00004.safetensors
Processing: model-00002-of-00004.safetensors
Processing: model-00004-of-00004.safetensors
Processing: model-00003-of-00004.safetensors
Processing files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:04<00:00,  1.21s/it]
 load weights over! 4858.21795463562 ms

<|im_start|>user
How are you<|im_end|>
<|im_start|>assistant
=================== start generate ====================



 Generation completed in 7519.09 ms
 Batchsize=1  Per_Batch_Input_Len=13  Per_Batch_New_Tokens=140

 Prefill TTFT: 0.86ms  Throughput: 15.12tok/s

 Decode  Avg ITL: 47.91ms   Throughput: 20.87tok/s

<think>
Okay, the user is asking "How are you?" which is a common greeting. I need to respond in a friendly and helpful way. Since I'm an AI, I can't actually feel emotions, but I should convey that I'm here to assist them. Maybe start with a positive greeting, then offer help. Let me make sure the response is concise and encourages them to ask for help if needed. Avoid any markdown formatting as per the instructions. Keep it natural and conversational.
</think>
Hello! I'm here to help you with any questions or tasks you have. How can I assist you today? 😊
total_time: 7561.41 ms

HCCL_PCIE_BUFFER_MODE=0 python examples/jiuge.py --metax --model_path=/data-aisoft/zenghua/models/9g_8b_thinking_llama/ --max_new_tokens=1024 --tp=2
Namespace(cpu=False, nvidia=False, metax=True, moore=False, iluvatar=False, cambricon=False, model_path='/data-aisoft/zenghua/models/9g_8b_thinking_llama/', max_new_tokens=1024, backend='cpp', batch_size=1, prompt='How are you', tp=2, enable_paged_attn=False)
 load weights ......
Processing: model-00001-of-00004.safetensors
Processing: model-00002-of-00004.safetensors
Processing: model-00004-of-00004.safetensors
Processing: model-00003-of-00004.safetensors
Processing files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:05<00:00,  1.28s/it]
 load weights over! 5123.946666717529 ms

<|im_start|>user
How are you<|im_end|>
<|im_start|>assistant
=================== start generate ====================



 Generation completed in 15601.74 ms
 Batchsize=1  Per_Batch_Input_Len=13  Per_Batch_New_Tokens=261

 Prefill TTFT: 1.25ms  Throughput: 10.42tok/s

 Decode  Avg ITL: 55.21ms   Throughput: 18.11tok/s

<think>
Okay, the user is asking "How are you?" which is a common greeting. I need to respond in a friendly and helpful way. Since I'm an AI, I don't have feelings, but I can still provide a positive and supportive answer.

I should start by acknowledging their question, then offer assistance. Maybe ask if they need help with something specific. Keep it concise but warm. Avoid being too formal or robotic. Use emojis if appropriate to make it more engaging.

Wait, the user might be testing if I'm capable of responding in a human-like manner. So, I should make sure my response is natural and not too stiff. Also, check if there's any cultural nuance I should consider. Since the query is in English, I'll keep the response in English.

Let me structure the response: greeting, offer help, invite further questions. Maybe add a friendly emoji at the end. Make sure it's clear and inviting. Avoid any technical jargon. Keep it simple and approachable.
</think>
Hello! I'm here to help you with any questions or tasks you have. How can I assist you today? 😊
total_time: 15652.59 ms

python test/bench/test_benchmark.py --metax /data-aisoft/zenghua/models/9g_8b_thinking_llama/ --bench ceval --subject middle_school_mathematics --num_samples 100 --backend cpp --ndev 1

============================================================
OVERALL RESULTS
============================================================
============================================================
Overall 成绩: 63/100 = 63.00%
Total Latency: 785.4413658697158 seconds
Total Tokens Processed: 32541 tokens
Overall Throughput: 41.43 tokens/s